Virtual Machine Trading in a Federation of Clouds: 
Individual Profit and Social Welfare Maximization 

Hongxing Li*, Chuan Wu*, Zongpeng LiT and Francis CM. Lau* 

*Department of Computer Science, The University of Hong Kong, Hong Kong, Email: {hxli, cwu, fcmlau}@cs.hku.hk 

TDepartment of Computer Science, University of Calgary, Canada, Email: zongpeng@ucalgary.ca 



cn 

o 

Oh 
< 






o 
en 



X 



Abstract — By sharing resources among different cloud 
providers, the paradigm of federated clouds exploits temporal 
availability of resources and geographical diversity of operational 
costs for efficient job service. While interoperability Issues across 
different cloud platforms in a cloud federation have been exten- 
sively studied, fundamental questions on cloud economics remain: 
When and how should a cloud trade resources (e.g., virtual 
machines) with others, such that its net profit is maximized over 
the long run, while a close-to-optlmal social welfare in the entire 
federation can also be guaranteed? To answer this question, 
a number of Important, inter-related decisions, including job 
scheduling, server provisioning and resource pricing, should 
be dynamically and jointly made, while the long-term profit 
optimallty is pursued. In this work, we design efficient algorithms 
for inter-cloud virtual machine (VM) trading and scheduling 
in a cloud federation. For VM transactions among clouds, we 
design a double-auction based mechanism that is strategyproof, 
individual rational, ex-post budget balanced, and efficient to 
execute over time. Closely combined with the auction mechanism 
is a dynamic VM trading and scheduling algorithm, which 
carefully decides the true valuations of VMs in the auction, 
optimally schedules stochastic job arrivals with different SLAs 
onto the VMs, and judiciously turns on and off servers based on 
the current electricity prices. Through rigorous analysis, we show 
that each individual cloud, by carrying out the dynamic algorithm 
in the online double auction, can achieve a time-averaged profit 
arbitrarily close to the offline optimum. Asymptotic optimallty 
in social welfare is also achieved under homogeneous cloud 
settings. We carry out simulations to verify the effectiveness of 
our algorithms, and examine the achievable social welfare under 
heterogeneous cloud settings, as driven by the real-world Google 
cluster usage traces. 

I. Introduction 

The emerging federated cloud paradigm advocates sharing 
of disparate cloud services (in separate data centers) from 
different cloud providers, and interconnects them based on 
common standards and policies to provide a universal environ- 
ment for cloud computing. Such a cloud federation exploits 
temporal and spatial availability of resources (e.g., virtual 
machines) and diversity of operational costs (e.g., electricity 
prices): when a cloud experiences a burst of incoming jobs, it 
may resort to VMs from other clouds with idle resources; when 
the electricity price for running servers and VMs is high at one 
cloud data center, the cloud can schedule jobs onto other cloud 
data centers with lower electricity charge at the moment. In 
this way, the aggregate job processing capacity of the cloud 
federation can be potentially higher than the aggregation of 
capacities of separate clouds operating alone, and the overall 
profit can be larger 

To implement the federated cloud paradigm, significant 
interest has arisen on developing interfaces and standards 



to enable cloud interoperability and job portability across 
different cloud platforms ( |T1 f2l)- However, fundamental 
problems on cloud economics remain to be investigated. A 
cloud in the real world is selfish, and aims to maximize its 
own profit, i.e., its income from handling jobs and leasing VMs 
to other clouds subtracting its operational costs and expenses 
in VM rental from other clouds. Only if its profit can be 
maximized and in any case not lower than when operating 
alone, can a cloud be incentivized to join a federation. This 
calls for an efficient mechanism to carry out resource trading 
and scheduling among federated clouds, to achieve profit 
maximization for individual clouds, as well as to perform well 
in social welfare. A number of inter-related, practical decisions 
are involved: (1) VM pricing: what mechanism should be 
advocated for VM sale and purchase among the clouds, and 
at what prices? (2) Job scheduling: with time-varying job 
arrivals at each cloud, targeting different resources and SLA 
requirements, should a cloud serve the jobs right away or later, 
to exploit time-varying electricity prices? And should a cloud 
serve a job using its own resources or others' resources? (3) 
Server provisioning: is it more beneficial for a cloud to keep 
many of its servers running to serve jobs of its own and from 
others, or to turn some of them down to save electricity? These 
decisions should be efficiently and optimally made in an online 
fashion, while guaranteeing long-term optimallty of individual 
cloud's profits, as well as the social welfare. 

In this paper, we design efficient algorithms for inter-cloud 
resource trading and scheduling, in a federation consisting of 
disparate cloud data centers. A double-auction based mecha- 
nism is proposed for the sell and purchase of available VMs 
across cloud boundaries over time. The auction is strategy- 
proof, individual rational, ex-post budget balanced, and com- 
putationally efficient (polynomial time complexity). Closely 
combined with the auction mechanism is an efficient, dynamic 
VM trading and scheduling algorithm, which carefully decides 
the true valuations of VMs to participate in the auction, 
optimally schedules randomly-arriving jobs with different re- 
source requirements (e.g., number of VMs) and SLAs (e.g., 
maximum job scheduling delay) onto different data centers, 
and judiciously turns on and off servers in the clouds based 
on the current electricity prices. The dynamic algorithm serves 
as an efficient strategy for each cloud to employ in the online 
double auction, and is proven to maximize individual profit for 
each cloud, over the long run of the system. The contributions 
of this work are summarized below. 

First, among the first in the literature, we address selfishness 
of individual clouds in a cloud federation, and design efficient 



mechanisms to maximize the net profit of each cloud. This 
profit is not only guaranteed to be larger than that when the 
cloud operates alone, but also maximized over the long run, in 
the presence of time-varying job arrivals and electricity prices 
at the cloud. 

Second, we novelly combine a truthful double auction 
mechanism with stochastic Lyapunov optimization techniques, 
and design an online VM trading and scheduling algorithm, for 
a cloud to optimally price the VMs and to judiciously schedule 
the VM and server usages. Each cloud values different VMs 
based on the back pressure in job queue scheduling, and bids 
them in the auction for effective VM acquisition. 

Third, we demonstrate that by applying the dynamic algo- 
rithm in the online double auction, each cloud can achieve 
a time-averaged profit arbitrarily close to its offline optimum 
(obtained if the cloud knows complete information on incom- 
ing jobs and electricity prices in the entire time span). We also 
prove that the social welfare, i.e., the time-averaged overall 
profit in the federation, can be asymptotically maximized when 
the number of clouds grows, under homogenous cloud settings. 
Trace-driven simulations examine the achievable social wel- 
fare with our dynamic algorithm under heterogenous settings. 

In the rest of the paper, we discuss related literature in 
Sec. ini present the system model in Sec. |III1 and introduce 
the detailed resource trading and scheduling mechanisms in 
Sec. |IV] A double auction mechanism is proposed in Sec. |V] 
and a benchmark social-welfare maximization algorithm is 
discussed in Sec. |VT] Theoretical analysis and simulation 
studies are presented in Sec. IVIII and Sec. IVIIII respectively. 
Sec. HX] concludes the paper 

II. Related work 
A. Optimal Scheduling in Cloud Systems 

Most existing literature (Il3]-|l6l and references therein) on 
resource scheduling in cloud systems focus on a single cloud 
that operates alone. A common theme is to minimize the 
operational costs (mainly consisting of electricity bills) in one 
or multiple data centers of the cloud, while providing certain 
performance guarantee of job scheduling, e.g., in terms of 
average job completion times Gl-lEI- 

Urgaonkar et al. |5| propose an algorithm with joint job 
admission control, routing and resource allocation for power 
consumption reduction in a virtualized data center. Rao et 
al. |3| advocate minimization of electricity expenses by ex- 
ploiting the temporal and spatial diversities of electricity 
prices. Yao et al. (|6] minimize the power cost with a two-time 
scale algorithm for delay tolerant workloads. Ren et al. pl also 
aim to minimize the energy cost while addressing the fairness 
in resource allocation. All the above works provide average 
delay guarantees for job services. 

Different from these studies on a stand-alone cloud with 
centralized control, this work investigates profit maximization 
for individual selfish clouds in a federation, where each par- 
ticipant makes its own decisions. Besides, bounded scheduling 
delay for each job is guaranteed even in worst cases, contrast- 
ing the existing solutions that ensure average delays. 



B. Resource Trading Mechanisms 

A rich body of literature is devoted to resource trading 
in grid computing fT\ and wireless spectrum leasing fSl ["91. 
Various mechanisms have been studied, e.g., bargaining [7|, 
fixed or dynamic pricing based on a contract or the supply- 
demand ratio 1 10 1, and auctions ["HI ["^l. 

A bargaining mechanism |7| typically has an unacceptable 
complexity by negotiating between each pair of traders. Fixed 
pricing, e.g., Amazon EC2 on-demand instances, has been 
shown to be inefficient in social welfare maximization in 
cases of system dynamics ifTTl . Dynamic pricing, such as 
Amazon EC2 spot instances, could be inefficient too, where 
the participants can quote the resources untruthfully ifTZI . 

Auction stands out as a promising mechanism, on which 
there have been abundant solutions (|8|, |9| and references 
therein) with truthful design and polynomial complexity. Al- 
though some recent works llTTI - llTSI aim to design an auction 
mechanism with individual rationality (non-negative profit 
gain) for trading in federated clouds, they do not explicitly 
address individual profit maximization over the long run, 
nor other desirable properties such as truthfulness, ex-post 
budget balance, and social welfare maximization. Moreover, 
little literature on auctions provides methods to quantitatively 
calculate the true valuations in each bid, which are simply 
assumed as known. Our design addresses these issues. 

III. System Model and Auction Framework 

A. Federation of Clouds 

We consider a federation of F clouds, each located at a 
different geometric location and operates autonomously to gain 
profit by serving its customers' job requests, managing server 
provisioning and trading resources with other clouds. 

Service demands: Each individual cloud i G [1,-F] has 
a front-end proxy server, which accepts job requests from 
its customers. There are S types of jobs serviced at each 
cloud, each specified by a three-tuple < irisjgsjds >. Here, 
nis S [Ij-'W^] specifies the type of the required VM instances, 
where M is the maximum number of VM types, and each 
type corresponds to a different set of configurations of CPU, 
storage and memory; gs is the number of type-ms VMs that 
the job needs simultaneously (See Amazon EC2 API |10]); 
and ds stands for the SLA (Service Level Agreement) of job 
type s e [1,5'], evaluated by the maximal response delay for 
scheduling a job, i.e., the time-span from when the job arrives 
to when it starts to run on scheduled VMs. In a cloud in 
practice, it is common to buy servers of the same configuration 
and provision the same type of VMs on one machine |14|. 
Therefore, we suppose each cloud i has N^^ homogenous 
servers to provision VMs of type m e [1, M], each of which 
can provide a maximum of C™ VMs of this type; the total 
number of servers in cloud i is Xlm=i ^i"*- 

The system runs in a time-slotted fashion. At the beginning 
of each time slot t, rf{t) e [0, Rf] jobs arrive at cloud i, for 
each job type s. Rf is an upper-bound on the number of type- 
s jobs submitted to cloud i in a time slot. The arrival of jobs 



is an ergodic process at each cloud. We suppose the arrival 
rate is given, and how a customer decides which cloud to use 
is orthogonal to this study. Let pf{t) G [0,p^ "'^ ] be the 
given service charge to the customer by cloud i, for accepting 
a job of type s in time slot t, which remains fixed within a 
time slot, but may vary across time slots. Here, j,^'-"*'^^' js the 
max possible price for pf{t). Such a general charging model 
subsumes pricing schemes in practice: e.g., time-independent 
p|(t) corresponds to the on-demand VM charging scheme, 
while time- varying pf (t) can represent the spot instance prices 
based on the current demand vs. supply fTOl. 

Job scheduling: Each incoming job to cloud i enters a FIFO 
queue of its type — a cloud i maintains a queue to buffer 
unscheduled jobs of each type s, with Qf{t) as its length in t. 
When the required VMs of a job are allocated, the job departs 
from its queue and starts to run on the VMs. A cloud may 
schedule its jobs on either its own VMs or VMs leased from 
other clouds, for the best economic benefits. Let ^fAt) be 
the number of type-s jobs of cloud i that are scheduled for 
processing in cloud j at the beginning of time slot t. 

When a job's demanded maximum response time (the SLA) 
cannot be met, in cases of system overload, it is dropped. 
A penalty is enforced in this case, to compensate for the 
customer's loss. Let 



Dt{t)e[Q,D: 



s{max)i 



(1) 



be the number of type-s jobs dropped by cloud i in t, where 



s{niax) 



jj^(,nu.j./ j^ ^j^^ maximum value of Df{t). Let ^| be the 
penalty to drop one such job, which is at least the maximum 
price charged to customers when accepting the jobs, i.e., 

j.^ ^ s(max) 
^!>P^ ■ 

Hence, the number of unscheduled jobs buffered at each 
cloud i E [l,F] can be updated with the following queueing 
law: 

F 

Qt{t + 1)= max{0^ (t) - J2 l^h (*) - Dt (t) , 0} 

+ rt{t), VsG[l,5]. (2) 

Job scheduling should satisfy the following SLA constraint: 
Each type-s job in cloud i is either scheduled or dropped (subject 

to a penalty) before its maximum response delay ds, Vs £ [1, S]. 

__ (3) 

We apply the e— persistence queue technique II15I . to create 

a virtual queue Zf associated with each job queue Qf (Vi G 

F 

Z^{t + 1) =max{Zr(t) + l{Q|(t)>o} ■ [es-Y,^^tJ{t)]-Dt{t) 

-l{QfM-o}-E ' . ' ,0},VsG[l,5]. (4) 
.=1 3' 



Here, ej > is a constant. 1 
indicator functions such that 

1 ifQ^(t)>0 



{Qf(«)>o} 



and 1 



{Q-(*)=o} 



=ni. are 



'■{Qf(t)>o} 



'■{Qf{*)=o} 



1 ifQ!it)=0 
Otherwise ' ^'^^^'^^'-"' IQ Otherwise 
Length of this virtual queue reflects the cumulated response 
delay of jobs from the respective job queue. Our algorithm 
seeks to bound the lengths of job queues and virtual queues. 



with properly set e^, and hence the maximum response delay 
of jobs can be bounded, i.e., constraint (O is satisfied. 

Server provisioning: We consider electricity cost, for running 
and cooling the servers |16|, as the main component of the 
operational cost in a cloud. Other costs, e.g., space rental and 
labour, remain relatively fixed for a long time, and are of less 
interest. Given that electricity prices vary at different locations 
and from time to time JS) ifTTl . we model the operational cost 
/3i(t) in each cloud i as a general ergodic process over time, 
varying across time slots between ^j™™^ and ^j™"-^^^ 

Each cloud strategically decides the number of active 
servers at each time, to optimize its profit. Let n"''{t) be the 
number of active servers provisioning type-m VMs at cloud i 
in t. The available server capacities at each cloud i G [l,F] 
constrain the feasible job scheduling at time t: 

J2 E 9sl^Ui) < Cr ■ nfit), Vm G [1, M], (5) 

je[l,F] s:m3=m,sg[l,Sl 

nr(i)<iVr, VmG[l,M]. (6) 

Q states that the overall demand for type-TTi VMs in cloud 
i from itself and other clouds should be no larger than the 
maximum number of available type-?Ti VMs on the active 
servers in cloud i. Here gsi^%{i) is the total number of VMs 
needed by type-s jobs scheduled from cloud j to cloud i in 
t. Motivated by practical job execution efficiency, we only 
consider scheduling a job to VMs from a single cloud, but 
not VMs across different clouds. (|6]l ensures that the number 
of active servers is limited by the total number of on-premise 
servers of the corresponding VM configuration at each cloud. 

B. Inter-cloud VM Trading with Double Auction 

In an inter-cloud resource market, VMs constitute the items 
for trading. For each type of VMs, multiple clouds may have 
them on sale while multiple other clouds can request them. A 
double auction is a natural fit to implement efficient trading in 
this case, allowing both selling and buying clouds to actively 
participate in pricing, on behalf of their own benefits. In our 
dynamic system, a multi-unit double auction is carried out 
among the clouds at the beginning of each time slot, deciding 
the VM trades within that time slot. 

Buyers & Sellers: A cloud can be both a buyer and a seller 
A buy -bid < 6™(i),7™(i) > records the unit price and 
maximum quantity at which cloud i is willing to buy VMs of 
type m, in t. Similarly, a sell-bid < s™ {t) , -q^^ {t) > records the 
unit price and maximum quantity at which cloud i is willing 
to sell VMs of type m in t. 

Let 6™(i) and s™(i) be cloud i's true valuation of buying 
and selling a type-m VM respectively (the max/min price it is 
willing to pay /accept). Similarly, let 7™(i) and ff^{t) be cloud 
i's true valuation of the quantity to buy and sell VMs of type 
m respectively (the maximum volume of VMs it is willing to 
purchase/sell). A cloud i may strategically manipulate the bid 
prices and volumes, in the hope of maximizing its profit. We 
show in Sec. I VIII that the double auction proposed in Sec. |IV] 
is truthful, such that each bid price reveals the true valuation. 
Auctioneer: We assume that there is a broker in the cloud 
federation, assuming the role of the auctioneer After collecting 



TABLE I 

Notation: input quantities and intermediate variables 



TABLE II 

Notation: decision variables at individual clouds 



F 


# of clouds 


S 


# of service types 


M 


# of VM types 


iris 


VM type of service type s 


ds 


Max. response delay of 
service type s 


9s 


# of VMs required by ser- 
vice type s 


rfit) 


# of type-s jobs arrived at cloud i, slot t 


H^ 


Max. # of type-s jobs arrived at cloud i per slot 


m) 


Service price for each job of type s at cloud i, slot t 


s{max) 
Pi 


Max. service price for each type-s job at cloud i per slot 


m) 


Cost for operating an active server at cloud i, slot t 


^(™>n, 


Min. cost for operating an active server at cloud i per slot 


„(max) 


Max. cost for operating an active server at cloud i per slot 


if 


Penalty for dropping a type-s job at cloud i 


^3(max) 


Max. # of type-s jobs cloud i drops per slot 


c™ 


Max. # of type-m VMs an active server at cloud i provisions 


Nr 


Total # of servers provisioning type-m VMs at cloud i 


Qt(t) 


Length of queue buffering type-s jobs at cloud i, slot t 


zm 


Length of virtual queue of type-s jobs at cloud i, slot t 


is 


Constant positive parameter for Zf(t), Vi G [l,i^] 


QS(max) 


Maximum length of queue Q| (t) 


yS[raax) 


Maximum length of virtual queue Zf (t) 


V 


User-defined constant positive parameter for dynamic algorithm 



all the buy and sell bids, the auctioneer executes a double 
auction to be detailed in Sec. |V] to decide the set of successful 
buy and sell bids, their clearing prices and the numbers of 
VMs to trade in each type. Let 6™(i) be the actual charge 
price for cloud i to buy one type-TTi VM, and 7™(t) be the 
actual number of VMs purchased. Similarly, let s™(i) be the 
actual income cloud i receives for selling one type-m VM, 
and 77™ (t) be the actual number of VMs sold. 

Let oi^it) be the number of type-m VMs that cloud i G 
[1, F] purchases from cloud j G [1, F] in t, as decided by the 
auctioneer: 

7r(i)= Yl <W, VmG[l,M], (7) 

CW= E <W, VmG[l,Ml. (8) 

J6[l,-Flj#i 

Since VMs are purchased for serving jobs, the job schedul- 
ing decisions i-ijJt) at each cloud i G [1, F], are related to the 
number of VMs it purchases: 

Yl gs-iA]{t) = aT]{t), 

VmG[l,M],VJ,ie[l,F],i7^j. (9) 

Three economic properties are desirable for the auctioneer's 
mechanism, (i) Truthfulness: Bidding true valuations is a 
dominant strategy, and consequently, both bidder strategies 
and auction design are simplified, (ii) Individual Rationality: 
Each cloud obtains a non-negative profit by participating in 
the auction, (iii) Ex-post Budget Balance: The auctioneer has 
a non-negative surplus, i.e., the total payment from all winning 
buy-bids is no less than the total charge for all winning sell- 
bids in each time slot. 

C. Individual Selfishness 

Each cloud in the federation aims to maximize its time- 
averaged profit (revenue minus cost) over the long run of 
the system, while striking to fulfill the resource and SLA 
requirements of each job. 



Mf,W 


# of type-s jobs scheduled from cloud i to cloud j, slot t 


nT(t) 


# of active servers providing type-m VMs at cloud i, slot t 


Df(t) 


# of dropped type-s jobs at cloud i, slot t 


srw 


True value of selling one type-m VM from cloud i, slot t 


vrw 


True value of volume to sel type-m VMs from cloud i, slot t 


srw 


Bid price for selling one type-m VM from cloud i, slot t 


vrw 


Max. # of type-m VMs cloud i can sell, slot t 


^rw 


True value of buying one type-m VM by cloud i, slot t 


7r(*) 


True value of volume to buy type-m VMs by cloud i, slot t 


bTW 


Bid price for buying one type-m VM by cloud i, slot t 


^r(t) 


Max. # of type-m VMs cloud i can buy, slot t 


table III 

Notation: decision variables at the auctioneer 


sTit) 


Actual price of selhng one type-m VM from cloud i, slot t 


firit) 


Actual # of type-m VMs sold from cloud i, slot t 


brit) 


Actual price of buying one type-m VM by cloud i, slot t 


iTW 


Actual # of type-m VMs bought by cloud i. slot t 


a™(t) 


Actual # of type-m VMs sold from cloud j to i, slot t 


ef(t) 


The j'" highest buy-bid price for type-m VMs at auctioneer 


&f(t) 


The j"^ lowest sell-bid price for type-m VMs at auctioneer 


Lf(t) 


Max. # of type-m, VMs to sell, in sell-bid with j"" lowest price 
at auctioneer in t 



Revenue: A cloud has two sources of revenue: i) job service 
charges paid by its customers, and ii) the proceeds from VM 
sales. The time-averaged revenue of cloud i G [l,i^] by 
undertaking different types of jobs from its customers is 

*i = Tiif^^E E nptit)-rt{t)}. (10) 

°° i=o 3e[i,s] 
We assume the front-end charges, pf{t), from a cloud to its 
customers, are given. Hence, this part of the revenue is fixed in 
each time slot. The time-averaged income of cloud i G [l,F] 
from selling VMs to other clouds is: 

*2 = ^ii^^E E nsTit)-vT{t)}. (11) 

t=0 me[l,M] 

Cloud i can control this income by adjusting its sell-bids, i.e., 
s™(i) and 7?™(i), Vm G [l,M], at each time. 

Cost: The cost of cloud i consists of three parts: i) operational 
costs incurred for running its active servers, ii) the penalties for 
dropping jobs, and iii) the expenditure on buying VMs from 
other clouds. The time-averaged cost for operating servers at 
each cloud i G [l,F] is decided by the number of active 
servers in each time, i.e., 

T-l M 

*i=t¥oo^E^{'3'W-E<W>- (12) 

t = m = l 

The time-averaged penalty at each cloud i G [1,F] is 
determined by the number of dropped jobs over time, i.e., 

Df{t), VsGil,^], iG [0,T-1]: 

T-l 



*', 



lim 



hJ2 J2 mt-Dt{t)}. 



T 



(13) 



=0 s6[l,S] 

The time-averaged expenditure for VM purchases is decided 
by the actual VM trading prices and numbers, as decided by 
the buy-bids (6^(i),7,"(i)) from cloud ie[l,F]: 

T-l M 



Profit Maximization: The profit maximization problem at 

cloud i G [l,F] can be formulated as follows: 

max $1 + $2 - *i - *2 - *3 (15) 

s.t. Constraints G}-©. 

D. Social Welfare 

Social welfare is the overall profit of the cloud federation: 

«e[i,F] 
Since the income and expenditure due to VM trades among 
the clouds cancel each other, the formula above equals 
J2ie\i F]i^i — ^1 — ^2)- The social welfare maximization 
problem is: 

max J2 (*i - *i - *2) (16) 

»6[1,F] 

S.t. Constraints Q-©, Vi £ [1,F] 
which globally optimizes server provisioning and job schedul- 
ing in the federation and maximally serves all the incoming 
jobs at the minimum cost, regardless of the specific inter-cloud 
VM trading mechanism. 

When a double auction mechanism is truthful, individual ra- 
tional and ex-post budget balancing, it is shown that efficiency 
in terms of social welfare maximization cannot be achieved 
concurrently [IS]. We hence make a necessary compromise in 
social welfare in our auction design, i.e., the sum of maximal 
individual profits derived by ( fTST i will be smaller than the 
optimal social welfare from ( fT&t . Nevertheless, we will show 
in Sec. IVIII and Sec. IVIIII that our mechanisms still manages 
to achieve a satisfactory social welfare in the long run. 

Tables H |ll] and |lll] summarize important notation in the 
paper, for ease of reference. 

IV. Dynamic individual-profit maximization 

ALGORITHM 

We next present a dynamic algorithm for each cloud to 
trade VMs and scheduling jobs/servers, which is in fact 
applicable under any truthful, individual-rational and ex -post 
budget balanced double auction mechanism. We will also tailor 
a double auction mechanism on the auctioneer in the next 
section. Fig. [T] illustrates the relation among these algorithm 
modules. 




Job scheduling & 

Server provisioning 
"Cloud i 




Fig. L Key algorithm modules. 

The goal of the dynamic algorithm at each cloud i is to max- 
imize its time-averaged profit, i.e., to solve optimization ( fTSl l. 
by dynamically making decisions in each time slot. We apply 
the drift-plus-penalty framework in Lyapunov optimization 
theory [191, and derive a one-shot optimization problem to be 
solved by cloud i in each time slot t as follows. We will prove 
in Sec. lVIII that by optimally solving the one-shot optimization 



at each cloud during each time slot, the dynamic algorithm can 
achieve a time-averaged individual profit arbitrarily close to its 
offline optimum (computed with complete knowledge in the 
entire time span), for each cloud. 

A. The One-shot Optimization Problem 

Define the set of queues at cloud i in each time slot t as 

Q^{t) = {Qm.Zt{t)\s^[l,S]}. 
Define the Lyapunov function as follows: 

L{e.{t)) = \ Y. [{Ql{t)f + {zt{t)f]. 

se[i,s] 

Then the one-slot conditional Lyapunov drift 11191 is 

A(e4t)) = L{Q,{t + 1)) - L(e.(i)). 

Squaring the queuing laws dU and (|4|i, we can derive the 
following inequality (details can be found in Appendix [All : 

HQ^(t))-v-[ Y K{t)vr{t)-bTmr{t)-Mt)nT{t)] 

?ne[l,M] 

se[i,s] 
<B,+ Y [QUt)rt{t) + Znt)es~VpUt)-rUt)] 

sS[l,S] 

~^m^<fim~^m, av) 

where V > is a user-defined positive parameter for 
gauging the optimality of time-averaged profit, Bi = 



[(Ef.ic: 
+ Ef-, ci 






+ D. 



s(7nax) \ 2 



r + (Rir + i^sY + 



2 2^s(_ 

^jjs(ma.) _!_ j.F^^ CY'NJ^'/gsf] is a constant, and 

v\{t) = v J2 isTmr{t)-bTmr{t)~Mt)nT{t)], 

mG[l,Af] 
s=S[1,S1jG[1,F] 

se[i,s] 

Based on the drift-plus-penalty framework 11191 . a dynamic 

algorithm can be derived for each cloud i, which observes the 
job and virtual queues (9i(i)), job anival rates {rf{t), Vs G 
[1, S]), the current cost for server operation i/3i{t)) in each time 
slot, and minimizes the RHS of the inequality ( fTTb . such that a 
lower bound for time-averaged profit of cloud i is maximized. 
NotethatB, + Eseli,s][Qmr!it) + Z!{t)e,-Vpm-rtit)] 
in the RHS of ( fTTb is fixed in time slot t. Hence, to maximize 
a lower bound of the time-averaged profit for cloud i, the 
dynamic algorithm should solve the one-shot optimization 
problem in each time slot t as follows: 

max ip\{t) + ^l{t) + iplit) (18) 

s.t. Constraints (TJ, ©-([g}. 
The maximization problem in ( fTSl l can be decoupled into 
two independent optimization problems: 

max (pl{t) + iplit) s.t. Constraints (|5}-(|9), (19) 

which is related to optimal decisions on i) buy/sell bids for 
different types of VMs, and ii) scheduling of active servers 
and jobs to these servers; and 

max fl{t) s.t. Constraint ([T), (20) 

which is related to optimal decisions on iii) jobs to drop. 
In the following, we design algorithms to derive the optimal 
decisions based on problem (fT9] l and problem 



B. VM Valuation and Bid 

Optimization problem ( fT9] l is related to the actual charges 
that cloud i pays for each type of VMs purchased, 6™ (t) and 
sf (t) (Vm e [1, M]), and the actual numbers of ti-aded VMs, 
7,'"(i) and 7}™(i) (Vm G [1,M]), from the double auction. 
These values are determined by the auctioneer according to 
buy-bids (6r(0:7l"(0) and sell-bids (sf'(i),?7,"(t)) submit- 
ted by all clouds, and its double auction mechanism. That 
is, each cloud i first proposes its buy-bids and sell-bids to 
the auctioneer, and then receives the auction results, based on 
which the job scheduling and server provisioning decisions are 
made. We first investigate how each cloud proposes its buy- 
bids and sell-bids, and then decide optimal job scheduling and 
server provisioning in Sec. IIV-CI 

A truthful double auction is employed at the auctioneer, 
where sellers and buyers bid their true values of the prices 
and quantities, in order to maximize their individual utilities. 
(fT9] l is the utility maximization problem for each cloud. If we 
can find true values of each cloud i, 6™(t),7™(<), s™(t) and 
?7™(i), and let the cloud bid using these values, the achieved 
utility in ( fT9] l is guaranteed to be the largest, as compared to 
bidding any other values. 

We decide the true values of the bids for each cloud i, 
according to their definitions in double auctions |8| [9|. The 
true value of the price to buy (sell) a type-m VM, 6™(t) 
(s™(i)), is such a value that, if a VM is purchased (sold) at a 
price (i) equal to this value, then cloud i's profit remains the 
same, compared to not obtaining the VM; (ii) higher than this 
value, a profit loss (gain) at cloud i occurs; and (iii) lower than 
this value, a profit gain (loss) results. In a multi-unit double 
auction, the true value of the maximum number of type-m 
VMs cloud i can buy (sell), 7j'"(i) (ry™(t)), is the maximum 
number of type-m VMs the cloud is willing to buy (sell) at 
the true value of the price, i.e., b™'{t) (s™(t)). 

Using the above rationale and based on problem ( fT9] l. the 
true values of the buy/sell prices for cloud i can be derived as 
(detailed derivation steps are ^iven in Appendix iBl i 



and 



bT{t) 



Q^"^it) + Z^^(t) 



V-gs 



(21) 



respectively, where 



Otherwise 



s*^ = arg max {W/ (i)}, 



and 



w:it) 



Qt it) + Zt (t) 



(22) 



(23) 



(24) 



Here, W^ (t) denotes the weight for scheduling one type-s' 
job (to run on type-TO^' VM(s)) by cloud i in t, and s*j 
specifies the job type with the largest weight (ties broken 
arbitrarily), among all types of jobs requiring type-m VMs. 
Wl (t) is determined by the following factors: (i) the sum 
of queue backlogs, Qf [t) + Zf (t), representing the level 
of urgency for scheduling type-s' jobs in t, since Qf (t) is 
the number of unscheduled type-s' jobs and Z^ (t) reflects 



the cumulated response delay; (ii) the number of concurrent 
VMs each type-s' job requires, gs', which decides the job- 
scheduling difficulty. 

The intuition behind (ISTT l and (|22] | includes: (i) the true 
value of the price to buy a type-m VM depends on the 
combined effect of urgency and difficulty for scheduling jobs 
requiring this type of VMs, and is computed based on the 
maximum weight that any type of jobs requiring type-m, VMs 
may achieve; (ii) the true value of the price to sell one type-m, 
VM from cloud i is the same as that of the price to buy, if 
the latter exceeds the current cost of operating a type-m, VM 
in the cloud; otherwise, it is set to the operational cost. 

The true values of the number of type-m VMs to buy and 
to sell at cloud i are 

7r(t)= J2 cT-Nr, (25) 

and #(tHCr-iVr, (26) 

respectively. They state that the maximum number of type- 
m, VMs cloud i is willing to buy (sell) at the price in (ISTT i 
(in (|22]|). is the number of all potential type-m, VMs in the 
federation. The rationale is as follows: The clearing price for 
transactions of type-m, VMs in the double auction is at most 
the buyer's true value in (ISTT i and at least the seller's true value 
in (l22l i. if the corresponding buy/sell bids are successful. By 
definition of the true value, if the actual charge per VM is 
lower (higher) than the true value, a profit gain happens at the 
buyer (seller), and the more VMs purchased (sold), the larger 
the profit gain. Therefore, a cloud is wilUng to buy or sell at 
the largest quantity possible, for profit maximizationl^ 

To conclude, in each time slot t, cloud i submits its bids as 
bf(t) = bf{t), s™(t) = sT(i), 7r(i) = lT\t) and ^(i) = 
?7,™(i), for each type of VMs m G [1, A/]. 

C. Server Provisioning, Job scheduling and Dropping 

After receiving results of the double auction (actual charges 
6™(t), s™(i), Vm G [1, M], and the actual numbers of traded 
VMs 7r(i), fiTii), Vm G [1,M], a™=(i),Vs G [l,5],Vj G 
[1,F]), cloud i schedules its jobs on its local servers and 
(potentially) purchased VMs from other clouds, decides job 
drops and the number of active servers to provision, by solving 
optimization problems (fT9] l and (l20l i. 

1) Server provisioning: We start with deriving n'"''{t), 
Vm, G [1,M], by assuming known values of s™(t), J7™(^)' 
6f'(t), 7™(i), a^(t) and nf^{t) (we wiU present the value of 
n™(t) in terms of these variables). In this case, problem (IT9^ 
is equivalent to the following minimization problem: 

min VA(t) J2 "^(*) 

?ne[l,M] 

s.t. Constraint (|5}, © and (|9). 

'it may appear counter-intuitive that a cloud is willing to buy all type- 
m VMs in the federation, regardless of its number of unscheduled jobs 
requirin g typ e-m VMs, i.e., '}2seli,S].ms=m Qfi^)- Interestingly, our proof 
in Sec. IVIII shows that bidding so in each time slot can achieve a time- 
averaged profit over the long run that approximates the offline optimum, 
and our simulation in Sec. IVIII I shows that it perfonns better as compared 
to a bidding strategy that asks for the exact number of VMs to serve the 
unscheduled jobs. 



Since V/3i (t) > 0, the best strategy is to assign the minimal 
feasible value to nY^{t), \/m £ [1, M], that satisfies constraints 
(|5]l and ^, which can be combined into 

Hence, the optimal number of activated servers at cloud i to 
provision type-m VM can be calculated as 

nT{t) = { Y. ^^li{t)■g. + Y.aJ,{t))/CT■ (27) 

These many servers can provide enough type-m VMs for 
serving local jobs and selling to other clouds. 

2) Job scheduling: We now derive iifj{t), Vj G [li-P]> 
s G [1, S], by assuming known values of s™(i), fj"^{t), V^it), 
7j"(i) and a^(i), with n™(t) given in Eqn. (E). Problem 
(fT9] l is equivalent to the following maximization problem: 

™^^ Y. Y. ^^U^)[QK^) + zt{t)] 

^'6[l,Sli6[l,F] 



s /,\ ys 



s.t. Constraint ([5}, ([6} and (|9). 
This is a maximum-weight scheduling problem, with 
Qf{t) + Z^{t) as the per-job scheduling weight for each /if ■ (t) 



(j ^i) and Ql{t) + Z!{t) 






as the per-job scheduling 



weight for each /ifj(i). There are two cases: 

> j — i: In this case, by combining constraints (|5]l, ^ and 
(|9]l, we have 

Based on the above maximum- weight problem, we know 
that the best strategy is to assign all the remaining type-m^ 
VMs in cloud i, C"='n"=(i) - Ej>^i""r(i) (the maximum 
number of on-premise type-m^ VMs minus those sold to other 
clouds), to serve its own jobs of service type s^ with the 



largest per-VM scheduling weight 



Ql{t)+z!{t) yft(t) 



9= c, 

n{t)+zt(t) -fOm+zHt) 



if it is 



positive (equivalently, the largest '^ ' — ^-^ if "^'^"^ ' "' '•"' > 
-£krr-), among all job types requiring type-rris VMs. Other- 
wise, cloud i does not serve any jobs using its own servers 
in t. Hence, we derive the optimal number of cloud i's type-s 
jobs scheduled to run on the cloud's local servers as 

Vs '* T. > -c^ 

and s = s^^ 
Otherwise 

(28) 
> j ^ i: ^lj{t) can be directly derived by q;™° (t), which is 

the number of type-?7is VMs cloud i purchased from cloud j 
(constraint (|5]i is satisfied by our server provisioning decision 
in Eqn. dZTl i. and constraint (|6]l is met by Eqn. ( l28T l and 
(|27]l), based on constraint (|9]l. Similar to the previous case, we 
know that the best strategy is to assign all the type-mg VMs 
purchased, a^," (t), to serve jobs of service type s*„^ with the 
largest per-VM scheduling weight 



Q'iW+z^jt) 



as defined in 



Eqn. (l23T l and JTM . Hence, we derive the optimal solution to 
the number of type-s jobs to run at cloud j(^ i) as 

'^"^^ ^ 1 Otherwise 



3) Job dropping: Problem (|20] i is a maximum-weight prob- 
lem with weight Qf{t) + Zfit) -— V ■ ^f for job-dropping 
decision variable Df{t), Vs S [1, 5], in the objective function. 
If the weight Qf(t) + Zf(t) - V ■ ^f > (i.e., if the 
level of urgency for scheduling type-s jobs Qf{t) + Zf{t) 
exceeds the weighted job-drop penalty V ■ ^f ), type-s jobs 
in queue Ql should be dropped at the maximum rate, i.e., 
Df{t) — D^y"^"-^)^ jjj order to maximize the objective function 
value; otherwise, there is no drop, i.e., Df{t) = 0. Therefore, 
the optimal number of type-s jobs dropped by cloud i in t is 

Otherwise 



Dt{t)^ 



(30) 



(29) 



In the above results, we note that the derived job scheduling 
and drop numbers do not need to be bounded by the number 
of unscheduled jobs in the corresponding job queue, i.e., 
UlAt) and -Df(i) are not required to be bounded by Qf(i) 
according to Eqn. (|2]i. Nevertheless, the actual number of 
jobs to schedule/drop when running the algorithm, is upper 
bounded by the length of the job queue. 

D. The Dynamic Algorithm 

Alg. [T] summarizes the dynamic algorithm for each cloud 
to carry out in each time slot, in order to maximize its time- 
averaged profit over the long run. 

Algorithm 1 Dynamic Profit Maximization Algorithm at cloud 
i in Time Slot t 

Input: rt{t), Qt{t), Z!{t), gs, rus, it, Cf, A^^ and /3>(t), Vs G 
[1,S]. 

Output: bT{t), sT{t), 7r(i), vnt), Dt{t), M?j(t) and nT{t), 
Vm G [1, M], sG [1,5], J G[1,F]. 
1: VM valuation and bid: Decide bTit), sTit), -y^it) and r^Tif) 

with Eqn. dlH-dlgl): 
2: Server provisioning, job scheduling and dropping: Decide 

litjit), Dl{t) and nTit) with Eqn. (jSjl, ^, ^ and ^■ 
3: Update Ql{t) and Zt(t) with Eqn. Q and &. 

We analyze the computation and communication complex- 
ities of Alg. [Has follows. 

Computation complexity: We study the computation com- 
plexity for each algorithm module respectively. 

i> VM valuation and bid: The algorithm should first calculate 
the value of s*„ for each VM type m G [1, M] with Eqn. ( l23l ) 
by comparing the weights Wf (t) among different types of 
jobs. In fact, the weight for each job type s G [1,5"] is only 
evaluated once since it is only involved in the calculation of 
s*j where m = mg. Hence, the computation overhead to find 
s*„, Vm G [1,M], is 0{S). Based on the value of s^, the 
buy/sell bids of type-m VMs can be decided by Eqn. (l2Tl i- 
( l26l l in constant time. For all M VM types, the computation 
overhead is 0{M). Hence, the overall computation complexity 
for this algorithm module is 0{S + M). 

> Server provisioning, job scheduling and dropping: With 
s*j, \/m G [1,A'I], calculated in the above algorithm module, 
we can directly know the value of s*^ , Vs G [1,5]. Then, 
the job scheduling decision nfJt) for job type s can be made 



in constant time based on Eqn. (1291 ) and ( |28] l. For all S job 
types, the computation overhead is 0{S). 

The server provisioning decisions can be found in constant 
time based on the job scheduling decisions and the auction 
results, according to Eqn. (127) for type-m VMs. For all M 
VM types, the computation overhead is 0{M). 

Job dropping is also decided in constant time for type-s 
jobs based on Eqn. ( l30l l. For all S job types, the computation 
complexity is 0{S). 

> Queue update: For each job type s, the job queue Ql{t) and 
virtual queue Zf (t) can be updated in constant time based on 
Eqn. (|2]) and (|4|i. Hence, for all S job types, the computation 
overhead is 0{S). 

In summary, the computation complexity of Alg.[T]is 0{S+ 
M). 

Coitununication complexity: The input to Alg. [T] is mostly 
derived from local information. There is no direct information 
exchange among individual clouds. The only communication 
overhead is incurred when a cloud sends its VM bids to the 
auctioneer and receives the auction results for each VM type. 
Since there are M VM types, the communication complexity 
is 0{M) for each cloud. 

V. Double Auction Mechanism 

We next design a double auction mechanism for inter-cloud 
VM trading, which not only is truthful, individual rational and 
ex-post budget balanced, but also can enable satisfactory social 
welfare (Theorems EH and |8] Sec. IVin . 

The true values of buy and sell bids at each participating 
cloud (Eqn. (|2T1|-(|26]|) are not related to the detailed auction 
mechanism. The true values of the maximum numbers of VMs 
a cloud is willing to trade (7™(t) and ?7™(t) in ( |25] ) and 
( |26] |) are time-independent constants determined by system 
parameters C™ and TV™. These parameters, and thus 7™(t) 
and ?/™(i), are easily known to other clouds, and hence it 
is not meaningful for a buyer/seller to bid otherwise. We 
correspondingly design a double auction where 7j'"(i) in each 
buy -bid is fixed to the value in ( |25] ) and 77™ {t) in each sell-bid 
is always the value in ( |26] |. while the buy/sell prices, b^^tys, 
and sf (i)'s, can be decided by the respective buyers/sellers. 

The following mechanism is carried out by the auctioneer at 
the beginning of each time slot t, to decide the actual trading 
price and number for each type of VMs me [1, M]. 
1. Winner Determination: The auctioneer sorts all received 
buy-bids for type-m VMs in descending order in the buy 
prices. Let 6']"(i) be the f^ highest. Two buy-bids with the 
largest and second largest prices, 9'^(t), 6'™(^)' ^^ identified 
(ties broken arbitrarily). The sell-bids for type-m VMs are 
sorted in ascending order in the sell prices. Let ?9™(t) be the 
j*'' lowest, with L™-{t) as the corresponding maximum num- 
ber of VMs to sell, such that d'^it) <-&T{t) < ...< i?^(i). 
Let j' be the critical index in the sorted sequence of sell-bids, 
such that ■d"}{t) is the largest sell price not exceeding 0'!J^{t), 
i.e., 

^]){t) < eT{t), and ^'P+,{t) > eT{t). (31) 



If there are at least two sell-bids iJ™ (0 ^^'^ ^T (0 "o higher 
than the second largest buy price 0™{t), the highest buy -bid 
6'™(t) wins, and the sell-bids with the lowest to the {j' — 1) 
lowest sell prices (7?™(t) < '&"l{t), not including j') win. 
Otherwise, no buy/sell bid wins. 

2. Pricing and Allocation: It is a NP-hard problem to clear 
the double auction market with discriminatory prices ll20l . We 
apply a uniform clearing price to winning buy/sell bids of 
type-TO VMs, as follows. 

> The price charged to each buyer cloud i of type-?7i VMs is 
'9Tit) ifbid far (i) wins, ^22) 

otherwise. 

t> The price paid to each seller cloud i of type-m VMs is 

'^7(t) if bid CW wins, ^2^^ 

otherwise. 

> The number of type-m VMs bought by cloud i is 



C(t) = 



sTit) = <! Q^' 



^rn (^^ ^ I J^S^ L7 (t) if bid bT (t) Wins, 
I otherwise. 



(34) 



The number of type-m VMs sold by cloud i is 
'■nrit) if bid sr(^) wins, 
otherwise. 

> The number of type-m VMs sold from cloud j to cloud i is 



vnt) = 



(35) 



a7]{t) = 



r7™(i) if bids &r(i) and sf(i) win, 



(36) 



otherwise. 

For example, consider a federation of 4 clouds with buy 
and sell prices bid in Table IIVI each seeking to buy/sell one 
VM. Clouds 2 and 3 bid the two largest buy prices $20 and 
$15, which are higher than sell prices from clouds 1 and 4. 
Hence the buyer cloud 2 and the seller cloud 4 win, while the 
clearing buy and sell prices are $15 and $13, respectively. 

TABLE IV 
Double auction bids : an illustrative example 





Cloud 1 


Cloud 2 


Cloud 3 


Cloud 4 


Buy-bid 


$10 


$20 


$15 


$8 


Sell-bid 


$13 


$22 


$16 


$9 



VI. Dynamic Social-Welfare Maximization 
Algorithm: a Benchmark 

We also present a dynamic algorithm that maximizes the 
time-averaged social welfare in the federation (optimization 
problem (O), and its derivation steps based on the Lyapunov 
optimization framework. This algorithm is used as a bench- 
mark to examine the efficiency of Alg. [T]in social welfare. 

A. Derivation Details 

Similar to the derivation of Alg. [T] we first derive a one-shot 
optimization problem (l40l i for the federation to solve based on 
the drift-plus-penalty framework of Lyapunov optimization, 
and then derive the dynamic benchmark algorithm to solve 
it optimally in each time slot. 

In each time slot t, define the set of queues 0(i) in the 
federation as 

e{t)^{Qt{t),ztme[i,F],se[i,s]}. 

Define the Lyapunov function as follows: 

Me(t)) = i J2 E [(Q'°W)' + (^'''W)'!- 

iS[l,F] se[l,S] 



Then the one-slot conditional Lyapunov drift is 

A(e(i)) = L(e(t + i))-L(e(t)). 

Squaring the queuing laws Eqn. (|2]i and (|4|i, we can derive 
the following inequality (details can be found in Appendix |Cj 

ie[i,Fi me[i,Af] se[i,s] 

- E pti^ydt)] 

s6[l,S] 

<B+ E E [Qt{t)rUt) + Znt)es-Vpt{t)rt{t)] 

iG[l,F] sS[l,Sl 

-Mt)~V2{i), (37) 

where V > is a user-defined positive parameter for 
gauging the optimality of the time-averaged social wel- 
fare, B = J2 



2 ^-^i 



\D\ 



G[1,S] 
s{max) I 



[E.^1 c; 



iG[l,-F] 



i?, is a constant with i?, = 









[i?; 



si 2 



E;=iCrA^fV3.]'],and 

^iW= E [ E [Qm + ztit)]- J2 ^^.w 

ie[i,F] se[i,s] je[i,-F] 

-VAW E ^^W]' (38) 

me[l,Af] 

^2W= E E Dtit)[Qt{t) + ZUt)^V-e,]. (39) 

ie[l,F] sG[l,S] 

Based on the drift-plus-penalty framework [19], a dynamic 
algorithm can be derived for the federation to observe job 
and virtual queues Q{t), job arrival rates {rf{t), Vi G 
[l,F],s e [1,'S']), the current cost for server operation 
{I3i{t), Vi e [Ij-f]) in each time slot, and minimizes the 
RHS of the inequality ( l37b . such that a lower bound for 
the time-averaged social welfare is maximized. Note that 

B + E.^[i,F] Es^[i,s][Qtityt(t) + z!{t)es - vptitynt)] 

in the RHS of ( [37] l is fixed in time slot t. Hence, to maximize 
a lower bound of the time-averaged social welfare for the 
federation, the dynamic algorithm should solve the one-shot 
optimization problem in each time slot t as follows: 

max (pi{t) + ip2{t) (40) 

s.t. Constraints Q, ^-^, Vi G [1,F]. 

The maximization problem in (|40] | can be decoupled into 
two independent optimization problems: 

max ipi{t) s.t. Constraint (H}-©, Vi G [1,F], (41) 
which is related to decisions on job scheduling and server 
provisioning, and 

max ip2{t) s.t. Constraint 0), Vi G [Ij-F], (42) 

which is related to decisions on job dropping. We note that 
to maximize social welfare, the decisions that the federation 
needs to make are not related to any inter-cloud VM trading 
mechanism, since the income and expenditure due to VM 
trades among the clouds have canceled each other when 
calculating the social welfare. We next solve problem (l4Tt 
and problem (|42] | to derive the optimal decisions. 

1) Server provisioning: We start with solving n"^{t), Mm G 
[1, Af], i G [1,F], by assuming known values of job schedul- 
ing decisions fifjtys, and present the value of the former in 
terms of the latter In this case, problem (l4ll is equivalent to 
the following minimization problem: 



min V Y. Mt) E "^W 

ie[l,F] mell,l\I] 

s.t. Constraint (gll - ((e}, Vi G [1, F]. 

Since V/3i{t) > 0, the best strategy is to assign the minimal 
feasible value to n^{t), for each VM type m at each cloud 
i, that satisfies constraints Q and (|6]l. Hence, the optimal 
number of activated servers at cloud i to provision type-TTi 
VMis 

^T{t)^[Yl E f^ut)-9s]/cr. (43) 

je[l,F] sS[l,S],m3 = m 

2) Job scheduling: We next derive fifJt), Vi G [l,i^], j G 
[1, F], s G [1, S"], with n™(i) given in Eqn. gS]). Problem gB 
is equivalent to the following maximization problem: 

"1^^ E E E t^m-[QUt) + z!{t)-vi3, 

i€[i,F]se{i,s]je[i,F] 
s.t. Constraint (H) - ©, Vi G [1, F]. 

This is a maximum-weight scheduling problem, with 

Qf{t) + Zl{t) 'gius as the per-job scheduling weight 

for each fJ,fj{t). Combining constraints (|5]i and (|6]l, we have 

E E gst^m<crNr,yj^[i,F]. 

ie[l,F] s:ms=m,sG[l,S] 

The best strategy is to assign all the type-m VMs in cloud j 
at the number of C^N"^ to serve jobs of type s^ of cloud i^ 
with the maximum per-VM scheduling weight 



C" 



\{t)+zi(t) 



Ijm^ ' if it is positive (equivalently, the largest '^ ' — ^-^ 



> 



as defined in Eqn. ^ and ^ in Sec. |IV]if QliM^UA 

J^ms ), among all job types from all clouds requiring type- 
m VMs. Hence, the optimal solution to the number of type-s 
jobs of cloud i to run at cloud j is 

M»j(*) = S and < j,s > = < w,im >, (44) 

[ Otherwise, 
where 

< im,Sm >= arg max {Wi{t)}, (45) 

ie[l,F],se[l,S],m3 = m 

and Wi{t) is the weight defined in Eqn. (l24l i. 

3) Job dropping: Problem ( |42] | is a maximum-weight prob- 
lem with weight Qf{t) + Zf{t) — V ■ £,f for job-dropping 
decision Df{t) in the objective function. If Ql{t) + Zf{t) — 
V • £,f > 0, type-s jobs at cloud i should be dropped at the 
maximum rate; otherwise, there is no drop. Hence, the optimal 
number of type-s jobs dropped by cloud i in f is 

I Otherwise. 

B. The Dynamic Benchmark Algorithm 

Alg.|2] summarizes the dynamic algorithm for the federation 
to carry out {e.g., on a centralized controller) in each time slot, 
in order to maximize its time-averaged social welfare over the 
long run. 
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Algorithm 2 Dynamic Social Welfare Maximization Algo- 

rithm in Time Slot t 

Input: rt{t), Ql{t), Z^(t), g,, m„ ^1, Cr, Nr and /3,(t), Vi G 
[l,i^],sG[l,S]. 

Output: Dt{t), fif^{t) and nT{t), Vi G [l,F],m G [l,M],s G 
[1,S]. 
1: Job scheduling and server provisioning: Decide ^lj{t) and 

nTit) with Eqn. (gill and (El; 
2: Job dropping: Decide Df{t) with Eqn. ( l46t : 
3: Update Qj'(i) and Zf (f) with Eqn. (IS and ^. 

VII. Performance Analysis 

We next analyze the performance guarantee provided by 
our dynamic individual-profit maximization algorithm and the 
double auction mechanism. 

A. Properties of the Double Auction Mechanism 

Theorem 1 (True Valuation): The VM valuations on buy- 
bids, i.e., Eqn. (ISTT i and (IZST l. and sell-bids, i.e., Eqn. (l22T i and 
(|26] |, are true values. 

This theorem is proved based on the definition of the true 
values and the optimization problem ( fTSl l solved in each time 
slot by each cloud in Appendix |B] 

Theorem 2 (Truthfulness): Bidding truthfully is the domi- 
nant strategy of each cloud in the double auction in Sec. [V] 
i.e., no cloud can achieve a higher profit in ([18) by bidding 
with values other than its true values of the buy and sell bids, 
in Eqn. (|2B(l25ll(|22li(|26ll. 

We prove this theorem by contradiction and show that, in 
all cases, no cloud can do better with problem (fTS) by bidding 
untruthfully. Details are in Appendix |D] 

Theorem 3 (Individual Rationality): No winning 

buyer pays more than its buy-bid price, and no 
winning seller is paid less than its sell-bid price, i.e., 
bTit) < bTit) and s^W > ^Tit), Mi G [l,i^],m G [l,M]. 

This theorem can be proved based on the winner determi- 
nation and pricing schemes in our auction mechanism, with 
details in Appendix |E] Given that the buy -bid (sell-bid) price 
is the true value of the buyer (seller), this theorem implies that 
a cloud can receive a non-negative profit gain, if it successfully 
sells or buys VMs. Hence, a cloud's profit obtained in a 
federation with potential VM trades with others, is always no 
lower than that obtained when operating alone. 

Theorem 4 (Ex-post Budget Balance): At the auctioneer, 
the total payment collected from the buyers is no smaller than 
the overall price paid to the sellers, i.e., 

E [^r(i) • lT{t) " -sTit) ■ f,T{t)] > 0, Vm G [1,M]. 
ie[i,Fl 
This theorem is proved based on Eqn. (ISTT l - (|33) . with 
details in Appendix |F] 



B. SLA Guarantee 



Lemma 1: Let * 



-^s(max) 



es- If D 



s{max) 



v^! 



-i?f andzf"'"''^ =Fef 



> inax{Rf,es}, each job queue Qf{t) and 



each virtual queue Zf (t) are upper-bounded by Q 



i(max) 



and 



This lemma can be proved by analyzing the job drop 
decision in ( l30l l and the queue updates in ©dill. The condition 
jjs(max) ^ niax{i?|, ts} ensurcs that, when the queue lengths 
grow to satisfy the job drop condition, any further increase 
on the queues, e.g., Rf and eg, can be balanced by dropping 
enough number of jobs at the rate of £)*'^'"°-^\ Detailed proof 
is included in Appendix iGl 

Theorem 5 (SLA Guarantee): Each job of type s G [1, S"] is 
either scheduled or dropped with Alg. [T] before its maximum 

response delay dg, if we set e^ = — 2 — ' ■ 

This theorem can be proved based on Lemma [T] and the 
e-persistence queue techniques ifTSl . The condition on ts is 
to ensure that the queue lengths can grow to satisfy the job 
drop condition, i.e., Q\ + Zf{t) > V£,f, if some jobs remain 
unscheduled in the last dg slots. Note that a cloud only drops 
jobs strategically, to balance the loss due to the job drop 
penalties and the gain in saving VMs for other jobs. For more 
details, please refer to Appendix |H] 

C. Optimality of Individual Profit and Social Welfare 

Theorem 6 (Individual Profit Optimality): Let fl* be the 
offline optimum of time-averaged profit of cloud i G [l,i^], 
obtained in a truthful, individual-rational, ex-post budget- 
balanced double auction, with complete information on its own 
job arrivals and prices in the entire time span [0,T — 1]. The 
dynamic Algorithm [T] can achieve a time-averaged profit Hi 
for cloud i within a constant gap Bi/V to Q,*, i.e., 

Qi > n* - B^/V, 

where F > and B, = ^ E.s[i,s][Ef=i C^^A^r/s. + 

a constant. 

The proof to this theorem is rooted in the Lyapunov op- 
timization theory [.19J . The gap Bi/V can be close to zero 
by fixing es and increasing V. Detailed proof is included in 
Appendix H] 

Theorem 7 (Social Welfare Optimality of Alg. |2|.- Let 11* 
be the offline optimum of the time-averaged social welfare in 
( fT6b . obtained with full information of the federation over the 
entire time span [0, T — 1]. The time-averaged social welfare 
achieved by all clouds by running Alg. |2] approaches the 
offline-optimal social welfare 11*, by a constant gap B/V, 
i.e., 

n > n* - B/V, 



E, 



e[l.F] 



Bi. Bi is defined in Theorem 



^s{raax] 



, respectively, in t G [0,r-l], Vi G [l,F],s G [1,5']. 



where V > Q and B 

m viG [i,F]. 

The proof to this theorem is also based on the Lyapunov 
optimization theory |19|. The gap B/V can be close to zero 
by fixing e^ and increasing V . Detailed proof is included in 
Appendix |j] 

Theorem 8 (Asymptotic Optimality in Social Welfare of Alg. Q}.- 
Let n* be the offline optimum of the time-averaged social 
welfare in (fTSl l. obtained with full information of the 
federation over the entire time span [0,r — 1]. Suppose all 
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clouds are homogenous, i.e., with the same number of servers 
(A^T") and the same maximum per-server VM provisioning 
(C™) for each VM type m, with i.i.d. service prices, job 
arrivals and operational costs. When the number of clouds, 
F, grows, the sum of time-averaged profits achieved by all 
clouds by running Alg. [T] under the double auction mechanism 
in Sec. [V] approaches the offline-optimal social welfare 11*, 
by a constant gap B/V, i.e., 

n > n* - B/V, 



E 



ie\l,F] 



Bi. Bi is defined in Theorem 



where V > Q and B 

ilVie [1,F]. 

To prove the theorem, we demonstrate that, when the 
number of clouds goes to infinity, the one-shot social welfare 
obtained with Alg. [T] is the same as that achieved by the 
dynamic benchmark Alg. |2]in the same time slot. Details are 
in Appendix iKl 

VIII. Performance Evaluation 
A. Simulation Setup 

We carry out trace-driven simulation studies based on 
Google cluster-usage data II2TI ll22l . which record jobs submit- 
ted to the Google cluster, with information on their resource 
demands (CPU, RAM, etc.) and relative charges. We translate 
the data into concrete job amval rates, resource types and 
prices, to drive our simulations as follows. 

We consider 24 types of jobs (< m^, g^, ds >), 6 VM types 
(nris) combined from {small, median, large} CPU and {small, 
large} Memory, and two SLA levels (ds), corresponding to 
a larger maximum respond delay and a smaller maximum 
response delay at half of the former Each job requires either 
1 VM or 2 VMs concurrently (cjs). 

There are 10 clouds in the federation. One time slot is 
1 hour The number of servers in each cloud that provision 
VMs of each type ranges within [800, 1000]. Each server can 
provide 30 small-memory VMs or 10 large-memory VMs. The 
VM charge to the customer is decided by multiplying gg by the 
relative VM price in the Google data, and then by the unit VM 
price in the range of [0.05, 0.08] $/h. The penalty for dropping 
a job is set to the maximum per-job VM charge in the system. 
Operational costs are set according to the electricity prices at 
10 different geographic locations provided in [17], which vary 
on a hourly basis. Each server consumes power at 1 KW/h. 

The number of job arrivals in each hour to the federation 
is set according to the cumulated job requests of each type 
submitted to the Google cluster during that hour, in the rough 
range of [40000, 90000] requests per hour We randomly assign 
each arrived job to one of the 10 clouds, following a heavy- 
tailed distribution. In operating the virtual queues, we set e^ = 
1000 for jobs requiring low response delay, and eg = 500 for 
those of long delays. The maximum number of job drops per 
hour is 1000 for all job types. 

For comparison purposes, we also implement a simpler 
heuristic algorithm for each cloud to bid in the double auction 
and to schedules its jobs/servers: The cloud decides a value 
for each unscheduled job in a queue as the penalty to drop 



it if the next time slot is the deadline for scheduling, or the 
charged price upon its arrival otherwise. The true values of 
buy/sell prices for a type-?7i VM at this cloud are set to the 
same, as the largest average value of jobs in a queue, among 
all job queues requiring type-m VM(s). The quantity of VMs 
in a buy-bid is set to the number of unscheduled jobs in the 
queue with the largest average value as computed above. The 
quantity of VMs in a sell-bid is the overall number of VMs 
of the type that the cloud can provide. All VMs purchased 
via the auction are used to serve jobs from the queue with 
the maximum average value. A cloud maintains the minimum 
number of servers to support those jobs, and only drops a job 
when its maximum response delay is reached. 
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Fig. 2. Comparisons of individual profit with different values of V. 

B. Individual Profit and Social Welfare 

We compare the time-averaged profit achieved at each cloud 
with our dynamic algorithm in Alg. [T] and with the heuristic 
algorithm, after the system has been running for 2000 hours. 
Fig. I2] shows that our algorithm can achieve a higher profit 
than the heuristic, at each of the 10 clouds, when the value of 
V is no less than 4 x 10*^. The observation is that when V is 
larger, the individual profit with our algorithm is even better, 
since it is closer to the offline optimum. 

We next compare the social welfare achieved with Alg. [T] 
the heuristic, and the dynamic benchmark Alg. |2] Fig. |3]shows 
that social welfare achieved with Alg. [T] is mostly within 
7.7% of that by the benchmark algorithm, even under our 
heterogenous settings. It outperforms the heuristic by 19.2%. 
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The social welfare is larger at larger V's in cases of both 
Alg. [T] and the benchmark algorithm, verifying Theorems |6] 
and |8] in that they approach the respective offline optimum 
when V grows. 




/(xlO") 

Fig. 3. Comparisons of social welfare. 

C. Response Delay and Job Drop 

We next investigate the scheduling delays experienced by 
jobs. In our system, a maximum response delay is set as 
the SLA objective for each type of jobs. Here, we study 
the average response delay actually experienced by the jobs, 
when the longer maximum response delay is set to different 
values. Fig. |4(a)| shows that both Alg. [T] and the benchmark 
algorithm incur a low average response delay (well ahead of 
scheduling deadlines), as compared to that of the heuristic. The 
reasons are: i) the heuristic algorithm always greedily keeps 
jobs in queues for future scheduling until near the deadline; 
and ii) both Alg. [T] and the benchmark algorithm evaluate the 
scheduling urgency better than the heuristic does, such that 
jobs are tended to be served well before the deadlines. 
-e-e-e o o o o ^ 
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Fig. 4. Comparisons of average job scheduling delay and drop percentage. 

We also study the percentage of admitted jobs in the entire 
federation that are eventually dropped with the three algo- 
rithms. Fig. |4(b)| reveals that the drop rate decreases quickly 
with the increase of the allowed maximum response delay, 
and Alg. [1] and the benchmark algorithm again outperform the 
heuristic, due to their well-designed scheduling strategies. 

IX. Conclusion 
This paper investigates both individual-profit maximizing 
and social-welfare efficient strategies at individual selfish 
clouds in a cloud federation, in VM trades across cloud 
boundaries. We tailor a truthful, individual-rational, ex -post 
budget-balanced double auction as the inter-cloud trading 
mechanism, and design a dynamic algorithm for each cloud 
to decide the best VM valuation and bidding strategies, and 
to schedule job service/drop and server provisioning in the 
most economic fashion, under time-varying job arrivals and 
operational costs. The proposed algorithm can obtain a time- 
averaged profit for each cloud within a constant gap to its 



offline maximum, as well as a close-to-optimum social welfare 
in the entire federation, based on both solid theoretical analysis 
and trace-driven simulation studies under realistic setting. As 
future work, we are interested in broadening our investigations 
to front-end job pricing and competition for customers among 
the clouds, and the connection between front-end charging 
strategies and inter-cloud trading strategies in a cloud federa- 
tion. 
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Appendix A 

Derivation of the one-shot optimization problem 

for individual profit maximization 

Squaring the queuing laws dU and dU, we can derive the 
following inequality 

sS[l,Sl j = i 
F 

- E^^^W - ^' W] + [lW|(*)>o}f=]' + [Dtit) 

j=i 

F 

j = l 

F 

j=i 
^^ E [E^r^r/3=+A^'™"^^]'+[^n' 

se[i,si i=i 

+ 2gnt)K(t)-E^»jW--^»W] 

F 

J=l 

F 

=B.+ E K»'WKw-E^^^w-^»'wi 

se[i,s] i=i 

F 

where B. = i E.s[i s^ll^U C^Np /9s+Df"'''-^r+lRt?+ 

By applying the drift-plus-penalty framework (or equiv- 
alently, drift-minus-profit here), we subtract the weighted 
one-shot individual profit of cloud i in time t, i.e., 

y ■ [E™e[i,M][^r(oc(i) - brmnt) - mxit)] + 

Eseli,s] iPtit) ■ ^tit) - Dt{t)m, on both sides of the above 
inequality. Hence, we have the following inequality: 

mG[l,Af] 

+ J2 [pUt)-rtit)-DUt)m] 

s6[l,S] 

<B.+ J2 [Qtit)r!{t) + Zt{t)e,~Vpt{t)-r!it)] 

se[i,s] 

-iplit) -ip\{t) -iplit), 

where 1/ > is a user-defined positive constant that can be 
understood as the weight of profit in the expression. 



Appendix B 
Finding true values 6f (i), 7r(i), s™(i) and 77™(i) 

Based on individual rationality and truthfulness of the dou- 
ble auction mechanism, each buyer/seller pays/charges a price 
that is no higher/lower than the corresponding bid (true) value, 
while the number of VMs actually traded is no larger than the 
bid (true) value if the bid is successful, i.e., V{^{t) < b™{t), 
sTit) > sT(i), irit) < iTit) and f,^{t) < fj^it). That 
is, the utility obtained by each cloud by participating in the 
auction is non-negative. Hence, the utility obtained by trading 
each VM at each winning buyer, i.e., b™'{t) — h'^{t), or seller, 
i.e., s™{t) — s™(i), is non-negative. Therefore, bidding for the 
maximum number of potential VMs provisioned, maximizes 
the utility of a seller or buyer, i.e., the maximum number of 
type-m VMs a cloud is willing to sell or buy is the maximum 
number of potential type-m VMs provisioned in the federation, 
and hence the true values of the VM volumes to bid at each 
cloud are derived as in Eqn. (l25t and (l2&t . respectively. 

We next identify the true values of the bidding prices for 
each type of VMs, m e [1, M], at cloud i case by case: 

> Case 1: Cloud i's buy-bid for type-m VMs wins, but not the 
sell-bid. 

In this case, we know that: i) all bought type-m VMs are 
from other clouds and should be used for job scheduling 
according to constraint (|9); ii) s™(t) = and Jy™(t) = 0. 

A nice property of problem ([19) is that, all decision vari- 
ables related to type-m VMs, i.e., fe^(i), 7™, 6™(t), 7,["(i), 
a^(t), n™'{t), and fifJt) with nis — m, are independent from 
those related to the other types of VMs. Hence, the optimal 
solutions to decision variables related to type-m VMs can be 
derived by solving the following sub problem from ([19): 

Y, J2 f^tjmQtit) + z:it)]-vbTmrit) 

s:ms=m,sg[l,Slie[l,Fl 



-VMt)nT{t) 
s.t. Constraint (|5)-(|9). 



(47) 



In ((47l), we replace 7™(i) by J2jeli,F]^'iji'^) based on 
Eqn. ((7), and replace /if (i)'s by the optimal solutions in 
Eqn. ((28) and ((25) (to be derived in Sec. [IV^ . We obtain 

aij{t)[— ^ Vbi (t)] 

j#»,ie[i,F] ^"^ 

+ l^lrmQf-it) + Zf-it)) ~ VMt)nT{t). (48) 
According to Eqn. ((28) and Eqn. ((27), if 9?'"W+^r'"W 



ft{t; 



Vg, 



> 



, we have 






Otherwise, we have 

P:r(i)(Q:"(i) + Zf-{t)) ~ Vm)nT{t) = 0. 

Both RHS values of the above equations are constants. As 
a result, the optimization problem ((48) is finally equivalent to 
the following one: 
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E 



^rn(^)[Q£:W + V:W _ VbTit)]. (49) 



According to the definition of true values, we know that 



the true value of b™'{t) should be 






as defined 



in Eqn. (EB, since: i) if fe™(i) > '^'"' ^vgf'"^ ^*^ ' ^^ utiHty in 
( |49] l is negative, and hence a profit loss in terms of problem 



^ for cloud i; ii) if 5™(t) < 






, the utility in 



(|49] l is positive, and hence a profit gain in terms of problem 

([T8]l; and iii) if 6™(<) = '^'"' yg^f''" ^*^ . the utihty in (Eg is 
zero, and the profit of cloud i in (jTsJ remains the same as not 
acquiring the VMs. 

> Case 2: Cloud i 's sell-bid for type-m VMs wins, but not the 
buy-bid. 

In this case, we know that: i) all type-m VMs sold from 
cloud i are used by other clouds for job scheduling according 
to constraint ©; and ii) S"(t) = and 7™(i) = 0. 

Similar to the analysis in Case 1, the optimal solutions to 
variables related to type-m VMs can be obtained by solving 
the following optimization problem: 

^ Y,iit,(t)[Qi{t) + zt{t)] + V3T{t)f,T{t) 



s.t. 



s:m.s— m,sG[l,5] j^i 

-VP,[t)nT[t) 
Constraint (O-©. 



(50) 



In ( l50l ). we replace 77™(i) by Y^j^i '^JiW based on Eqn. ^ 
and the fact in this case that a™(i) = 0, and replace /i.f^(t)'s 
and n™(0's with the optimal solutions in Eqn. ( |28] |, ( l29l l and 
Eqn. (IZTI i. Then we have the following two cases: 



i)if 



^9aS, 



> c^' problem 

'"(t) + zfr(t) 



Nricr 



is equivalent to 



Q:-it) + z;-'{t). 



9< 



'(t) + Z '"(*) 



where the true value of s™(t) should be y — 

according to the definition of true value of the price to sell a 

type-m, VM; 

ii) otherwise, problem (BOl l is equivalent to 

max vJ2aT^it)[ST{t) - Mt)/Cr], 

where the true value of s™(i) should be (ii{t)/CY^ according 
to the definition of the true value of the price to sell a type-m 
VM. Hence, we have derived the true values of s™(t) given 
in Eqn. ( |22] |. 

l> Case 3: Both Cloud i 's buy-bid and sell-bid for type-m VMs 
win. 

In this case, the following properties hold: 

Property 1. If both cloud i's buy-bid and sell-bid for type-m 
VMs win, the cloud cannot buy a type-m VM with a price 
strictly higher than its price to sell a type-m VM, i.e., sl"{t) > 
b"''{t). Otherwise, there will be a positive profit loss at the 



cloud by self-trading its own type-m, VMs, which violates its 
individual rationality. 

Property 2. If both cloud i's buy-bid and sell-bid for type- 
m, VMs win, the cloud cannot sell a type-m VM with a price 
strictly higher than its price to buy a type-m VM, i.e., s™(t) < 
hY^{t). Otherwise, the auctioneer has to pay a positive sum to 
compensate for the price difference for those inter-cloud traded 
type-m, VMs, which contradicts the ex-post budget balance 
property at the auctioneer 

Property 3. Based on Properties 1 and 2, if both cloud i's 
buy-bid and sell-bid for type-vn VMs win, the actual buy and 
sell prices at the cloud for type-m, VMs are the same, i.e.. 

According to Property 3, we derive that the overall profit 
gain at cloud i for self-trading of type-m VMs, a™ (t) [s™ (t) — 
&™(t)], is zero. 

The optimal solutions to variables related to type-m VMs 
can be obtained by solving the following optimization prob- 
lem: 



J2 E ^^tJmQKt) + znt)] 

s:m3=m,sS[l,Sl iG[l,F] 



+ vsT{t)fjrit)' 

s.t. Constraint (|D-(|9). 



VMt)nT{t) 



vbTmnt) 



(51) 



In (|5B, we replace ^^{t) by X]jg[i _f] o^ij (0 based on 
Eqn. 0, and rif^{t) by EiG[i,F] ai?(0 based on Eqn. dUl. We 
also replace pfjtys and n™(i)'s with the optimal solutions 
in Eqn. ( l28T l. ( |29] l and Eqn. (|27] |. Then we have the following 
two cases: 

p,„-, problem jSTT i is equivalent to 



jN jf q;"'(*)+^i'"(*) > iM 



Nric: 



E"™w[- 



^{t)+z:-{t) 



VP^{t)] 



rrit)+z:-it) 



J7^! 



9s^ 



VbTit)] 



+ Y,a]l{t)lVsT{t) 



>:"w+^:'"(t), 



where the true values of 6f (i) and s™(i) should both be 

^ — y — ^- — -^ according to the definition of true value of the 

price to buy/sell a type-m VM; 

ii) otherwise, problem (ISTT l is equivalent to 



max y^ a^j (i) [ 



Qi-w+^r-ft) 



i#i 



9s* 



VbTit)] 



and 



vj2<msT{t)-Mt)/cr], 

where the true value of 6™(i) should be — — y ^ ' — —^, 
the true value of S™(t) should be j3i{t)/CY^ according to the 
definition of the true value of the price to buy/sell a type-m, 
VM. Hence, we have derived the true values of &™(i) and 
s™(i) given in Eqn. ( 1211 1 and Eqn. ( |22] |. 
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Appendix C 

Derivation of the one-shot optimization problem 

for social welfare maximization 

Squaring the queuing laws (|2]l and dUl, we can derive the 
following inequality 

A(e(i))<i J2 E [Ea'^^W + ^-^WI' + I^'^WI' 

i6[l,F] se[l,S] j=l 
F 

F 



j=i 



+ l{Q?(t)>0}E^^j(*)] 



2Z^{t)[l 



j=i 



iQ!{t)>o} 



[e^-J2f^Ui)]-Dtit) 



i=i 



^{QtW=o}J2^T'^j 



j=i 



^^ E E [[EcrA^r/ff^+A^'™'']' + [^n 



i6[l,F] sS[l,S] j=l 



+ 2Q|(t)K(f)-^/i^,(f)-A'(f)] 



+ h]' + [D, 



j=i 



i(max) 1^ X ^ /-ims 






2zf(i)[6.-EM^.w-A^w]i 



ie[i,F] se[i,s] j=i 

F 

+ Zt{t)[es -Y^iiUt) - Dt{t)]l 
where B = I E-e[i,^, E.6n,s][[Ef=i C^^iV-Vp. + 

By applying the drift-plus-penalty framework (or 
equivalently, drift-minus-profit here), we subtract the 
weighted one-shot social welfare in time t, i.e., 

y ■ E,;6[i,i.]E™e[i,M][^r(0^r(i) - &r(i)7r(0 - 

A:(iXW] + E.e[i,S][PlW • rm - Omm^ on both 
sides of the above inequality. Hence, we have the following 
inequality: 

A(e(i))-\/- Y. [ E [sTmT{t)-hTmT(t) 

iS[l,Fl me[l,Ml 

-A(t)nr(t)]+ E [ptit)-n{t)-Dt{m]] 

se[i,s] 

<B+ E E [Qt{t)rKt) + Zt{t)e,~Vpt(t)Tt{t)] 

ie[i,F] se[i,si 
-(^i(i) -(^2(i), 

where 1/ > is a user-defined positive constant that can be 
understood as the weight of profit in the expression. 



Appendix D 
Proof to Theorem|2] 

This theorem can be proved based on the following two 
lemmas. 

Lemma 2 (Monotonic winner determination): Given prices 
of buy-bids {&5"(i), . . . ,&f'(t), . . . ,6^(t)} and sell-bids 
{s5"(t), . . . , s™(t), . . . , s'j}{t)}, we have that 

1) If cloud i wins the buy -bid by bidding with b'"''{t), 
then cloud i also wins the buy-bid by bidding with 
b' > b'^{t); 

2) If cloud i wins the sell -bid by bidding with s™(i), 
then cloud i also wins the sell-bid by bidding with 
s' < sTit). 

Proof: We prove the cases in the lemma respectively as 
follows, 

1) Since cloud i wins the buy -bid with b™'{t), we know that 
&™(t) is the largest among all buy -bids, i.e., b"''{t) > 
b^{t), Vj ^ i,j e [1,F]. With b' > 6^"(i), we have 
that b' > 6J'(i), Vj ^ i,j e [1,F]. Hence, if cloud i 
proposes a buy-bid with b', it can still win the buy-bid 
according to our winner determination decision, since 
its buy-bid price is still the largest among all buy-bids. 

2) Since cloud i wins the sell-bid with s™(i) and the 
sell-bids are sorted in ascending order, we know that 
s™(0 ^ i^i"(^)' where j' is the critical index as defined 
in Eqn. (l3ll l. and s™(t) is among the (j' — 1)*'* lowest 
sell-bids in ascending order. With s' < s™(i), we also 
have that s' < ■Q'^{t) and s' is among the {f — 1)*'* 
lowest sell-bids in ascending order Hence, if cloud i 
propose a sell-bid with s' , it can still win the sell-bid 
according to our winner determination decision, since 
its sell-bid price is still among the [f — 1)*'' lowest 
sell-bids. 

■ 
Lemma 3 (Bid-independent pricing): Given prices of 
buy-bids {6f (t), . . . ,67'(i), . . . ,6^(t)} and sell-bids 
{s5"(i), . . . , s™(t), . . . , s^(i)}, we have that 

1) If cloud i wins the buy -bid by bidding with b™'{t) and 
6', the charged price 6™(i) to cloud i is the same for 
both; 

2) If cloud i wins the sell-bid by bidding with s™ (t) and 
s', the charged price s™(t) to cloud i is the same for 
both. 

Proof: We prove the the cases in the lemma respectively 
as follows, 

1) Since cloud i wins the buy-bid, the charged price 5™(i) 
should be 9^{t), which is the second largest buy-bid 
price independent of cloud i's buy-bid, according to our 
pricing scheme in the auction. And we have that 6™ (t) > 
B'^{t) and b' > Oipit). We also have that, as long as 
cloud i wins the buy-bid, the value of the second largest 
buy -bid price 0™{t) does not change. Hence, cloud i 
should be charged with 0™ (0 ^y bidding with no matter 
bY'{t) or b'. 
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2) Since cloud i wins the sell-bid, the charged price s™ (t) 
should be ^"?(t), which is the j'*'* lowest sell-bid price, 
according to our pricing scheme in the auction. And we 
have that s™(t) < '!9™(t) and s' < ?9J/(i). As long as 
cloud i wins the sell-bid, we have that s"'-{t) and s' 
should be among the {j' — 1)*'' lowest sell-bid prices 
and the value of 'd"}{t) does not change. Hence, cloud 
i should be charged with the same price with i?"?(i) by 
bidding with no matter 6™(i) or b' if it wins the sell-bid. 

■ 
We can then prove the Theorem |2] to show that any cloud 
i cannot obtain higher utility gain by bidding untruthfully, 
i.e., b"^{t) ^ 6™(t) and/or s™(i) ^ S™(i), Vm e [l,Af], 
by analyzing all possible auction results. 
Case 1 - Cloud i wins both buy-bid and sell-bid with truthful 
bidding: In this case, the charged/paid prices for buy-bid and 
sell-bid are 62'' (t) and ■d"f{t), respectively. We discuss the all 
possibly cases of untruthful bidding as follows, 

. Bid untruthfully with h^{t) > bf'it) and sf^(i) > Sf^(i): 
According to Lemma |2] cloud i still wins the buy-bid 
but may either win or lose the sell-bid. If it also wins 
the sell-bid, we have that the charged/paid prices for 
buy -bid and sell-bid are 0''2'{t) and 'd"}{t), respectively, 
according to Lemma |3j the utility gain is zero by bidding 
untruthfully since the charged/paid prices are the same 
with that by bidding truthfully. If it loses the sell-bid, 
we have that the charged prices for buy-bid and sell-bid 
are 0™(i) and 0, respectively, according to Lemma |3] and 
our pricing scheme, and no VM is sold by cloud i; the 
utility gain is non-positive by bidding untruthfully, since 
the charged buy-bid price remains the same while there is 
non-negative utility loss, i.e., \&'^{t) — s™{t)\f)™{t) with 
^"■Kt) > sr(0' by losing the sell-bid. 

. Bid untruthfully with b'^'{t) > b'^'{t) and s™(i) = S™(i): 
According to Lemma |2] cloud i still wins the buy-bid. 
It is also easy to see that cloud i still wins the sell- 
bid with the same bidding price for sell-bid. Hence, the 
charged/paid prices for buy-bid and sell-bid are 6*™ (t) and 
i?"?(i), respectively, according to Lemma |3] The utility 
gain is zero by bidding untruthfully. 

. Bid untruthfully with 6™(t) > 6™(t) and s\''{t) < S™(i): 
According to Lemma|2] cloud i still wins both the buy-bid 
and the sell-bid. The charged/paid prices for buy-bid and 
sell-bid are 9™{t) and i?"/(t), respectively, according to 
Lemma|3] The utility gain is zero by bidding untruthfully. 

. Bid untruthfully with b'^{t) ^ b'^{t) and sf'it) > S^(i): 
Cloud i stills wins the buy-bid with the same bidding 
price for buy-bid. However, it may either win or lose 
the sell-bid. If it also wins the sell-bid, we have that the 
charged/paid prices for buy-bid and sell-bid are 0™(i) 
and i?"/(t), respectively, according to Lemma |3] and our 
pricing scheme; the utility gain is zero by bidding un- 
truthfully since the charged/paid prices are the same with 
that by bidding truthfully. If it loses the sell-bid, we have 
that the charged prices for buy -bid and sell-bid are 0™ (t) 



and 0, respectively, according to our pricing scheme, and 
no VM is sold by cloud i; the utility gain is non-positive 
by bidding untruthfully, since the charged buy-bid price 
remains the same while there is non-negative utility loss, 
by losing the sell-bid. 

Bid untruthfully with b^(t) = 6™(i) and s'^it) < S™(t): 
According to Lemma |2] cloud i still wins the sell-bid. 
It is also easy to see that cloud i still wins the buy- 
bid with the same bidding price for buy-bid. Hence, the 
charged/paid prices for buy-bid and sell-bid are 9™ (t) and 
•d'"}{t), respectively, according to Lemma |3] The utility 
gain is zero by bidding untruthfully. 
Bid untruthfully with b'^{t) < b"^\t) and sf{t) > 5™(t): 
The cloud can either win both buy and sell bids, or 
win buy-bid only, or win sell-bid only, or lose both 
bids. If the cloud still wins both bids, we have that the 
charged/paid prices for buy-bid and sell-bid are 0™(i) 
and i?'"(i), respectively, according to our pricing scheme; 
the utility gain is zero by bidding untruthfully since the 
charged/paid prices are the same with that by bidding 
truthfully. If it only wins the buy-bid, we have that the 
charged prices for buy -bid and sell-bid are ^"(i) and 
0, respectively, according to Lemma our pricing scheme, 
and no VM is sold by cloud i; the utility gain is non- 
positive by bidding untruthfully, since the charged buy- 
bid price remains the same while there is non-negative 
utility loss, by losing the sell-bid. If it only wins the 
sell-bid, we have that the charged prices for buy-bid and 
sell-bid are and i9"?(t), respectively, according to our 
pricing scheme, and no VM is bought by cloud i; the 
utility gain is non-positive by bidding untruthfully, since 
the charged sell-bid price remains the same while there 
is non-negative utility loss, i.e., [fc™(i) — 9^2' (t)] ■ 7i"(^) 
with 6™(t) > 9l''{t), by losing the buy-bid. If it loses 
both bids, we have that the charged prices for buy-bid and 
sell-bid are both 0, according to our pricing scheme, and 
no VM is bought by or sold by cloud i; the utility gain is 
non-positive by bidding untruthfully, since there is non- 
negative utility loss, i.e., [bTit)-9l''{t)]-j^{t) + [&J}{t)- 
sTit)] ■ vTit) with 6^(i) > e^{t) and i?™(i) > Sf^(i), 
by losing the both bids. 

Bid untruthfully with b^'{t) < b^'{t) and s^^it) = S™(t): 
Cloud i stills wins the sell-bid with the same bidding 
price for sell-bid. However, it may either win or lose the 
buy-bid. If it also wins the buy-bid, we have that the 
charged/paid prices for buy -bid and sell-bid are 02"(^) 
and ^^'"(i), respectively, according to our pricing scheme; 
the utility gain is zero by bidding untruthfully since the 
charged/paid prices are the same with that by bidding 
truthfully. If it loses the buy-bid, we have that the 
charged prices for buy-bid and sell-bid are and ■d'^it), 
respectively, according to our pricing scheme, and no 
VM is bought by cloud i; the utility gain is non-positive 
by bidding untruthfully, since the charged sell-bid price 
remains the same while there is non-negative utility loss. 
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by losing the buy-bid. 
. Bid untruthfully with b'^{t) < b'^{t) and sf'(i) < Sf'(i): 
According to Lemma |2] cloud i still wins the sell-bid 
but may either win or lose the buy-bid. If it also wins the 
buy-bid, we have that the charged/paid prices for buy-bid 
and sell-bid are ^"(t) and ^"f{t), respectively, according 
to Lemma [3] and our pricing scheme; the utility gain is 
zero by bidding untruthfully since the charged/paid prices 
are the same with that by bidding truthfully. If it loses 
the buy-bid, we have that the charged prices for buy-bid 
and sell-bid are and i?"^(t), respectively, according to 
Lemma |3] and our pricing scheme, and no VM is bought 
by cloud i; the utility gain is non-positive by bidding 
untruthfully, since the charged sell-bid price remains the 
same while there is non-negative utility loss, by losing 
the buy-bid. 

Case 2 - Cloud i wins buy-bid but loses sell-bid with truthful 
bidding: In this case, the charged/paid prices for buy-bid and 
sell-bid are ^"(t) and 0, respectively, we discuss the all 
possible cases of untruthful bidding as follows, 

. Bid untruthfully with 6™(t) > 6™(t) and s'^{t) > Sf^(i): 
According to Lemma |2] cloud i still wins the buy-bid. 
It is also easy to see that cloud i still loses the sell-bid, 
since otherwise we will have a contradiction to Lemma 
|2] Hence, the charged/paid prices for buy-bid and sell-bid 
are 9^^{t) and 0, respectively, according to Lemma |3] and 
our pricing scheme. The utility gain is zero by bidding 
untruthfully. 

. Bid untruthfully with b^(t) > b^(t) and sl^it) = 5^(i): 
According to Lemma |2] cloud i still wins the buy-bid. 
It is also easy to see that cloud i still loses the sell- 
bid with the same bidding price for sell-bid. Hence, the 
charged/paid prices for buy-bid and sell-bid are 0™{t) 
and 0, respectively, according to Lemma |3] and pricing 
scheme. The utility gain is zero by bidding untruthfully. 

. Bid untruthfully with b'^{t) > b'^{t) and sf{t) < SY'it): 
According to Lemma |2l cloud i still wins the buy -bid but 
may either win or lose the sell-bid. If it loses the sell- 
bid, we have that the charged prices for buy-bid and sell- 
bid are 0™(i) and 0, respectively, according to Lemma |3] 
and our pricing scheme, and no VM is sold by cloud i; 
the utility gain is zero by bidding untruthfully since the 
charged/paid prices are the same with that by bidding 
truthfully. If it also wins the sell-bid, we have that the 
charged/paid prices for buy-bid and sell-bid are 0^(i) 
and ?9",'(i) < s™(i), respectively, according to Lemma |3] 
and our pricing scheme; the utility gain is non-positive 
by bidding untruthfully, since the charged buy-bid price 
remains the same while there is non-negative utility loss, 
i.e., [S™(i) - ■&]}{t)]fil^it) with i9^(t) < S™(i), by 
winning the sell-bid. 

. Bid untruthfully with b^(t) = b^(t) and s'^{t) > s'^it): 
Cloud i still wins the buy-bid with the same bidding price. 
It is also easy to see that cloud i still loses the sell-bid, 
since otherwise we will have a contradiction to Lemma |2] 



Hence, the charged/paid prices for buy-bid and sell-bid 
are 6'™(i) and 0, respectively, according to our pricing 
scheme. The utility gain is zero by bidding untruthfully. 

. Bid untruthfully with bY'it) = b"^\t) and sf{t) < 5™(t): 
Cloud i still wins the buy-bid with the same bidding 
price. However, it may either win or lose the sell-bid. 
If it loses the sell-bid, we have that the charged prices 
for buy-bid and sell-bid are 0™(i) and 0, respectively, 
according to our pricing scheme, and no VM is sold by 
cloud i; the utility gain is zero by bidding untruthfully 
since the charged/paid prices are the same with that by 
bidding truthfully. If it also wins the sell-bid, we have 
that the charged/paid prices for buy-bid and sell-bid are 
0^{t) and ■d"}{t) < s™(i), respectively, according to our 
pricing scheme; the utility gain is non-positive by bidding 
untruthfully, since the charged buy-bid price remains the 
same while there is non-negative utility loss, by winning 
the sell-bid. 

. Bid untruthfully with b^(t) < 6™(i) and s^(i) > s™(t): 
It is easy to see that cloud i still loses the sell-bid, since 
otherwise we will have a contradiction to Lemma |2] How- 
ever, it may either win or lose the buy-bid. If it also wins 
the buy-bid, we have that the charged/paid prices for buy- 
bid and sell-bid are 9^^{t) and 0, respectively, according 
to our pricing scheme; the utility gain is zero by bidding 
untruthfully since the charged/paid prices are the same 
with that by bidding truthfully. If it loses the buy-bid, we 
have that the charged prices for buy-bid and sell-bid are 
and 0, respectively, according to Lemma our pricing 
scheme, and no VM is bought or sold by cloud i; the 
utility gain is non-positive by bidding untruthfully, since 
the charged sell-bid price remains the same while there 
is non-negative utility loss, i.e., [6™(i) - 6*2" (i)] • 7™(i) 
with 6™(t) > e^'{t), by losing the buy-bid. 

. Bid untruthfully with 6™(t) < 6™(i) and s^^it) = 5™(t): 
The cloud i still loses the sell-bid with the same bidding 
price. However, it may either win or lose the buy-bid. If 
it also wins the buy-bid, we have that the charged/paid 
prices for buy -bid and sell-bid are 0™(i) and 0, respec- 
tively, according to our pricing scheme; the utility gain is 
zero by bidding untruthfully since the charged/paid prices 
are the same with that by bidding truthfully. If it loses 
the buy-bid, we have that the charged prices for buy-bid 
and sell-bid are and 0, respectively, according to our 
pricing scheme, and no VM is bought or sold by cloud 
i; the utility gain is non-positive by bidding untruthfully, 
since the charged sell-bid price remains the same while 
there is non-negative utility loss, by losing the buy-bid. 

. Bid untruthfully with b'p{t) < 6™(i) and s™(t) < sY'it): 
The cloud can either win both buy and sell bids, or 
win buy-bid only, or win sell-bid only, or lose both 
bids. If the cloud wins both bids, we have that the 
charged/paid prices for buy-bid and sell-bid are 02"(^) 
and i9"?(i) < s™(0, respectively, according to our pricing 
scheme; the utility gain is zero by bidding untruthfully 
since the charged buy-bid price remains the same while 
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there is non-negative utility loss, by winning the sell- 
bid. If it only wins the buy-bid, we have that the 
charged prices for buy -bid and sell-bid are 0^^{t) and 
0, respectively, according to our pricing scheme, and 
no VM is sold by cloud i; the utility gain is zero by 
bidding untruthfully since the charged/paid prices are the 
same with that by bidding truthfully. If it only wins 
the sell-bid, we have that the charged prices for buy- 
bid and sell-bid are and 'd"}{t) < s^{t), respectively, 
according to our pricing scheme, and no VM is bought 
by cloud i; the utility gain is non-positive by bidding 
untruthfully, since there is non-negative utility loss, i.e., 

[bTit) - OTit)] ■ iTit) - [i>'p{t) - ~sT{t)] ■ iiTit) with 

6™(t) > ei^{t) and ?9J/(i) < 5™(t), by losing the buy-bid 
while winning the sell-bid. If it loses both bids, we have 
that the charged prices for buy-bid and sell-bid are both 0, 
according to our pricing scheme, and no VM is bought 
by or sold by cloud i; the utility gain is non-positive 
by bidding untruthfully, since the charged sell-bid price 
remains the same while there is non-negative utility loss, 
i.e., [hT{i)~0^'{i)]-lT{t) with&7^(t) > 6*^" (i), by losing 
the buy-bid. 

Case 3 - Cloud i wins sell-bid but loses buy-bid with truthful 
bidding: In this case, the charged/paid prices for buy-bid and 
sell-bid are and 'd"}{t), respectively, we discuss the all 
possible cases of untruthful bidding as follows, 

. Bid untruthfully with b'Pit) > 6™(t) and s^(i) > S^(i): 
The cloud can either win both buy and sell bids, or win 
buy-bid only, or win sell-bid only, or lose both bids. If 
the cloud wins both bids, we have that the charged/paid 
prices for buy-bid and sell -bid are 0'!p{t) > h™{t) and 
•d'"}{t), respectively, according to our pricing scheme; the 
utility gain is non-positive by bidding untruthfully since 
the charged sell-bid price remains the same while there is 
non-negative utility loss, i.e., [fiTl^-^rC*)] •%'"(*) with 
^™(0 ^ ^™(0' by winning the buy-bid. If it only wins 
the buy-bid, we have that the charged prices for buy-bid 
and sell-bid are 6'™(t) and 0, respectively, according to 
our pricing scheme, and no VM is sold by cloud i; the 
utility gain is non-positive by bidding untruthfully since 
there is non-negative utility loss, i.e., [62'' (t) — b^it)] ■ 



Tit) + mit) 



'(i)] . frit) with fe™(t) < 0™(t) 



and 'd"}{t) > sY^{t), by winning the buy -bid while losing 
the sell-bid. If it only wins the sell-bid, we have that the 
charged prices for buy -bid and sell-bid are and -d") (t) < 
s™(i), respectively, according to our pricing scheme, and 
no VM is bought by cloud i; the utility gain is zero by 
bidding untruthfully since the charged/paid prices are the 
same with that by bidding truthfully. If it loses both bids, 
we have that the charged prices for buy-bid and sell-bid 
are both 0, according to our pricing scheme, and no VM 
is bought by or sold by cloud i; the utility gain is non- 
positive by bidding untruthfully, since the charged buy- 
bid price remains the same while there is non-negative 
utility loss, i.e., [d"}{t) - s™(t)] ■ 77f'(i) with i9"(t) > 



s™(i), by losing the sell-bid. 

. Bid untruthfully with b^{t) > b^{t) and s'^{t) = 5™(t): 
Cloud i still wins the sell-bid with the same bidding price. 
However, it may either win or lose the buy-bid. If it also 
wins the buy-bid, we have that the charged/paid prices for 
buy -bid and sell-bid are 0™{t) and i?'"(i), respectively, 
according to our pricing scheme; the utiUty gain is non- 
positive by bidding untruthfully since the charged sell- 
bid price remains the same while there is non-negative 
utility loss, i.e., [e^{t) - &f^(i)] • 7™(i) with 6^(i) < 
0'^{t), by winning the buy -bid. If it loses the buy -bid, 
we have that the charged prices for buy-bid and sell-bid 
are and 'd"}{t), respectively, according to our pricing 
scheme, and no VM is bought or sold by cloud i; the 
utility gain is non-positive by bidding untruthfully, since 
the charged/paid prices are the same with that by bidding 
truthfully. 

. Bid untruthfully with b'^{t) > b^{t) and s'^'{t) < S^{t): 
According to Lemma |2] cloud i still wins the sell-bid but 
may either win or lose the buy -bid. If it loses the buy-bid, 
we have that the charged prices for buy-bid and sell-bid 
are 9^{t) and i?"?(i), respectively, according to Lemma 
[3] and our pricing scheme, and no VM is sold by cloud i; 
the utility gain is zero by bidding untruthfully since the 
charged/paid prices are the same with that by bidding 
truthfully. If it also wins the buy-bid, we have that the 
charged/paid prices for buy -bid and sell-bid are 0™ [t] > 
6™(i) and ^"^{t), respectively, according to Lemma |3] 
and our pricing scheme; the utility gain is non-positive 
by bidding untruthfully, since the charged sell-bid price 
remains the same while there is non-negative utility loss, 
by winning the buy-bid. 

. Bid untruthfully with b^{t) = b^{t) and sf'it) > S™(t): 
Cloud i still loses the buy-bid with the same bidding 
price. However, it may either win or lose the sell-bid. If 
it loses the sell-bid, we have that the charged prices for 
buy-bid and sell-bid are and 0, respectively, according 
to our pricing scheme, and no VM is sold by cloud i; the 
utility gain is non-positive by bidding untruthfully since 
the charged buy-bid price remains the same while there 
is non-negative utility loss, i.e., [i?"?(i) — s^{t)] ■ J)™(i) 
with 'd"}{t) > s™(^)' by losing the sell -bid. If it also wins 
the sell-bid, we have that the charged/paid prices for buy- 
bid and sell-bid are and d"}{t) < s™(i), respectively, 
according to our pricing scheme; the utility gain is zero 
by bidding untruthfully since the charged/paid prices are 
the same with that by bidding truthfully. 

. Bid untruthfully with b'^{t) = 6™(i) and sf{t) < 5™(t): 
According to LemmalU cloud i still wins the sell-bid. It is 
also easy to see that cloud i still loses the buy-bid with 
the same bidding price. Hence, the charged/paid prices 
for buy-bid and sell-bid are and i9"^(t), respectively, 
according to Lemma[3]and our pricing scheme. The utility 
gain is zero by bidding untruthfully. 

. Bid untruthfully with 6™(<) < 6™(i) and s'^{t) > §"^{1): 
It is easy to see that cloud i still loses the buy-bid. 
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since otherwise we will have a contradiction to Lemma |2] 
However, it may either win or lose the sell-bid. If it wins 
the sell-bid, we have that the charged/paid prices for buy- 
bid and sell-bid are and d"i{t), respectively, according 
to our pricing scheme; the utility gain is zero by bidding 
untruthfully since the charged/paid prices are the same 
with that by bidding truthfully. If it loses the sell-bid, we 
have that the charged prices for buy-bid and sell-bid are 
both 0, according to our pricing scheme, and no VM is 
bought or sold by cloud i; the utility gain is non-positive 
by bidding untruthfully, since the charged buy-bid price 
remains the same while there is non-negative utility loss, 
by losing the sell-bid. 

. Bid untruthfully with b'^{t) < b''^{t) and s'^^t) = .?™(i): 
Cloud i still wins the sell-bid with the same bidding price. 
It is also easy to see that cloud i still loses the buy-bid, 
since otherwise we will have a contradiction to Lemma ID 
Hence, the charged/paid prices for buy-bid and sell-bid 
are and ^"?(i), respectively, according to our pricing 
scheme. The utility gain is zero by bidding untruthfully. 

. Bid untruthfully with b^(t) < b^(t) and s^^^t) < S^(i): 
According to Lemma |2] cloud i still wins the sell-bid. 
It is also easy to see that cloud i still loses the buy-bid, 
since otherwise we will have a contradiction to Lemma 
|2] Hence, the charged/paid prices for buy-bid and sell-bid 
are and ??'"(i), respectively, according to Lemma |3] and 
our pricing scheme. The utility gain is zero by bidding 
untruthfully. 

Case 4 - Cloud i loses both buy-bid and sell-bid with truthful 
bidding: In this case, the charged/paid prices for buy-bid 
and sell-bid are both 0. we discuss the all possible cases of 
untruthful bidding as follows, 

. Bid untruthfully with b'^'{t) > 67^ (i) and s'Pit) > Sf'(i): 
It is easy to see that cloud i still loses the sell-bid, 
since otherwise we will have a contradiction to Lemma 
|2] However, it may either win or lose the buy-bid. If it 
wins the buy-bid, we have that the charged/paid prices 
for buy-bid and sell-bid ai-e e^'{t) > b^{t) and 0, 
respectively, according to our pricing scheme; the utility 
gain is non-positive by bidding untruthfully there is non- 
negative utility loss, i.e., [0'^{t) - 6f'(i)] • 7,'"(i) with 
6™(t) < e^{t), by winning the buy-bid. If it loses the 
buy-bid, we have that the charged prices for buy-bid and 
sell-bid are both 0, according to our pricing scheme, and 
no VM is bought or sold by cloud i; the utility gain is 
zero by bidding untruthfully. 

. Bid untruthfully with b'^{t) > b'^{t) and sf{t) = Sf'(i): 
Cloud i still loses the sell-bid with the same bidding price. 
However, it may either win or lose the buy-bid. If it loses 
the buy-bid, we have that the charged prices for buy-bid 
and sell-bid are both 0, according to our pricing scheme, 
and no VM is sold by cloud i; the utility gain is zero 
by bidding untruthfully. If it also wins the buy-bid, we 
have that the charged/paid prices for buy-bid and sell-bid 
are 0™(^) — ^"'(0 ^^^ 0' respectively, according to our 



pricing scheme; the utility gain is non-positive by bidding 
untruthfully there is non-negative utility loss, by winning 
the buy-bid. 

Bid untruthfully with b'^{t) > b"^\t) and sf{t) < 5™(t): 
The cloud can either win both buy and sell bids, or win 
buy-bid only, or win sell-bid only, or lose both bids. If 
the cloud wins both bids, we have that the charged/paid 
prices for buy-bid and sell-bid are 0™(i) > b^{t) and 
^]^!'{t) < S™(t), respectively, according to our pricing 
scheme; the utility gain is non-positive by bidding un- 
truthfully since there is non-negative utility loss, i.e., 

Kit) - bTit)] ■ iT{t) + [^Tit) - ^"(i)] • f}T{t) with 

6™(i) < 6'^'(i) and i?J/(i) < S™(t), by winning the buy- 
bid and sell-bid. If it only wins the buy-bid, we have 
that the charged prices for buy -bid and sell-bid are 02"(^) 
and 0, respectively, according to our pricing scheme, and 
no VM is sold by cloud i; the utility gain is non-positive 
by bidding untruthfully since the paid price for sell-bid is 
the same with that by bidding truthfully and there is non- 
negative utility loss, by winning the buy-bid. If it only 
wins the sell-bid, we have that the charged prices for buy- 
bid and sell-bid are and i?"?(i) < s™(i), respectively, 
according to our pricing scheme, and no VM is bought 
by cloud i; the utility gain is non-positive by bidding 
untruthfully since the charged price for the buy-bid is the 
same with that by bidding truthfully while there is non- 
negative utiHty loss, i.e., [5f'(i) - ^'p[t)\ ■ f}Y\t) with 
i?^(i) < Sr(^)' by winning the sell-bid. If it loses both 
bids, we have that the charged prices for buy-bid and 
sell-bid are both 0, according to our pricing scheme, and 
no VM is bought by or sold by cloud i; the utility gain 
is zero by bidding untruthfully. 

Bid untruthfully with b^'{t) = 6™(i) and s'^'{t) > 5™(t): 
It is easy to see that cloud i still loses the sell-bid, since 
otherwise we will have a contradiction to Lemma |2] It 
is also not hard to find that cloud i still loses the buy- 
bid with the same bidding price. Hence, the charged/paid 
prices for buy-bid and sell-bid are both 0, according to 
our pricing scheme. The utility gain is zero by bidding 
untruthfully. 

Bid untruthfully with b^(t) = 6™(i) and s^^it) < S™(t): 
It is not hard to find that cloud i still loses the buy- 
bid with the same bidding price. However, it may either 
win or lose the sell-bid. If it wins the sell-bid, we have 
that the charged/paid prices for buy-bid and sell-bid are 
and ■d'"}{t) < s™(i), respectively, according to our 
pricing scheme; the utility gain is non-positive by bidding 
untruthfully since the charged price for buy-bid is the 
same with that by bidding truthfully while there is non- 
negative utility loss, by winning the sell-bid. If it loses 
the sell-bid, we have that the charged prices for buy-bid 
and sell-bid are both 0, according to our pricing scheme, 
and no VM is bought or sold by cloud i; the utility gain 
is zero by bidding untruthfully. 
Bid untruthfully with b^'{t) < b^'{t) and s''{'{t) > 5™(t): 
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It is easy to see that cloud i still loses both the buy- 
bid and the sell-bid, since otherwise we will have a 
contradiction to Lemma|2] Hence, the charged/paid prices 
for buy-bid and sell-bid are both 0, according to our 
pricing scheme. The utility gain is zero by bidding 
untruthfully. 

. Bid untruthfully with 6™(t) < 6™(t) and sf'it) = S™(i): 
It is easy to see that cloud i still loses the buy-bid, since 
otherwise we will have a contradiction to Lemma |2] It 
is also not hard to find that cloud i still loses the sell- 
bid with the same bidding price. Hence, the charged/paid 
prices for buy-bid and sell-bid are both 0, according to 
our pricing scheme. The utility gain is zero by bidding 
untruthfully. 

. Bid untruthfully with b'^{t) < b'^{t) and sf{t) < Sf'(i): 
It is easy to see that cloud i still loses the buy-bid, 
since otherwise we will have a contradiction to Lemma |2] 
However, it may either win or lose the sell-bid. If it wins 
the sell-bid, we have that the charged/paid prices for buy- 
bid and sell-bid are and i}"}{t) < s™(^)' respectively, 
according to our pricing scheme; the utility gain is non- 
positive by bidding untruthfully since the charged price 
for buy-bid is the same with that by bidding truthfully 
while there is non-negative utility loss, by winning the 
sell-bid. If it loses the sell-bid, we have that the charged 
prices for buy-bid and sell-bid are both 0, according to 
our pricing scheme, and no VM is bought or sold by 
cloud i; the utility gain is zero by bidding untruthfully. 

To conclude, we have shown that bidding truthfully is the 
dominant strategy of each cloud. 

Appendix E 
Proof to Theorem[3] 

We prove the individual rationality for buy-bids and sell- 
bids, respectively. 

Winner of buy-bid: If cloud i wins the buy-bid for VM type 
m, we have that 6™(i) is the largest buy-bid price among 
all buy -bids, and 6™(t) > 6'™(i), according to our winner 
determination scheme. Since K"(i) = 62'' (t) according to our 
pricing scheme, we have that 6™(i) < h^it). 
Winner of sell-bid: If cloud i wins the sell-bid for VM type 
m, we have that s™(f) is among the (j' — 1)*'* lowest sell-bid 
prices of all sell-bids, and s'l^if) < ■d^.p{t) according to our 
winner determination scheme. Since s™(t) = ^"?(i) according 
to our pricing scheme, we have that s™(i) > s^{t). 

Appendix F 
Proof to Theorem|4] 

We first calculate the total payment from the buyers and the 
total price paid to the sellers, respectively. We then show that 
the ex-post budget balance is guaranteed. 
Total payment from the buyers: According to our winner 
determination scheme, only the buyer with largest bidding 
price wins the buy-bid. With the pricing scheme in Eqn. (l32l l. 
the charged price is 0'^{t) for each bought VM. With the 
allocation scheme in Eqn. ( |34] |. the overall number of bought 



VMs is X]7=i ^j"(^)- Hence, the total payment from the 
buyers is that 

^ [6r(t)-7rw] = er(i)-E^rw- 

ie[l,F] j = l 

Overall price paid to the sellers: According to our winner 
determination scheme, only the seller with (j' — 1)*'' lowest 
bidding price wins the sell-bid. With the pricing scheme in 
Eqn. ( |33] |. the paid price is i9"?(i) for each sold VM. With 



the allocation scheme in Eqn. (1351 ), the overall number of sold 
VMs is X)7=i ^Ti't)- Hence, the total payment to the sellers 
is that 

ie[l,F] j = l 

According to Eqn. dSTT i for the winner determination 
scheme, we know that ^'"(i) < 6'™(0- Hence, the ex-post 
budget balance at the auctioneer for each VM type m G [1, M] 
can be guaranteed as follows, 

»S[1,F] 

= E K{t)-ir{t)]- E isT{t)-vr{t)] 

iG[l,F] ie[l,F] 

=9T{t) ■ E LTit) - ^rit) ■ E LTii) 
>0. 

Appendix G 
Proof to Lemma[T] 

We prove the lemma by induction. 

Induction Basis: At time slot 0, the beginning of the federa- 
tion, all queues are empty. Therefore, 

QtiO) = < Q^(™"-), Vi G [1, F], s G [1, S], 

Z-{0) = < Z,'=(™-), Vi G [1, F], s G [1, S]. 
Induction Step: Suppose that, at time slot t > 0, Qf{t) < 

for any Qf (t) and Z^{t), we have the following possible cases. 

. < Qm < v(! or vi! < Qtit) < yef + RV. 

. < zf (i) < Fef or y^l < Z!{t) < y^i + £,. 

- We first analyze the size of Qf{t + 1): 
. If < Ql{t) < V£,f, we have that 

F 

Qt{t+1) = ma^{Qt(t) - E^'.W " Dt (t) , 0} + rt (t) 
<max{Qt{t),0} + Rt 

according to the queueing law (|2]i. The first inequality is 
based on the fact that < rf (i) < Rf. 
. If V^f < Qi{t) < V^l + Rf, we have that 

according to the job drop decision with Eqn. (|30] |. 
Hence, we have that 



s{max) 



The second inequality is based on the fact that jjp™"-^' > 



Qt{t + 1) = maxjQf (t) - ^ Mf,(t) - A'(t), 0} + r^i) 

< maxinr + ^1 - D^^(™"', 0} + Rt 

< ve, + R! < 

lini 

max{i?f,es}. 

So far, Ql{t) < Q^^(""^), \/i e [l,F],s e [1,5*] for each 
time slot t is proved. 
- We next analyze the size of Zf{t + 1): 

. If < Zf{t) < V£,f, we have that 

F 

Zi{t+1) =max{Zi'(t) + l{Q=(t)>o} • [es -^^^^(t)] - D ■ 



Zr (r + 1) = maxjZr (r) + ^s - ^ Mz, (r) - Dt (r), 0} 



j=i 



j=i 



<ycr + es = ^f^™""', 

according to the queueing law (|4]i. The first inequality is 
based on the fact that e^ > 0. 

If V^f < Zl{t) < V^l + es, we have that 

according to the job drop decision with Eqn. (|30] |. 
Hence, we have that 

F 

Z-{t+l) = max{Zi{t) + l{Qj(t)>o} • [e^ -^fj-tjit)] - -D- 



Summing the above over r G [t + 1, . . . , t + d^], we have 
that 

t + ds F 

Z^i + d. + 1) - Z!{t + 1) > 6.d, - I] E ^^^■(^) + -O* (^)]- 

T = t + 1 j = l 

Rearranging the above inequality and using the fact that 
Z^{t + ds + l) < Zl^"""""^ and Zfit + 1) > 0, we have that 

(f) e.d. - Z;(™'"^U Y.[Y,^,l,{r) + Dt(T)]. (52) 

T = t + 1 j = l 

Since the jobs are scheduled in a FIFO fashion, the jobs 
r? (i) that arrive at slot t are placed at the end of the queue at 
slot < + 1, and should be fully cleared only when all the jobs 
backlogged in Qf (t+ 1) have been scheduled. That is, the last 
job of rl {t) departs on slot t + T with T > as the smallest 

integersatisfyingEttT+i[Ef=iM.f;(T)+AnT)] > Ql{t+l). 
Based on our assumption that not all of the rl{t) jobs depart 
by time t ^ dg, we must have that 



(i) 



Combining Eqn. ( l52b and ( l54b . we have that 

^ s - Z^ < Qi 



siraax) 



j=i 



>es < 



rfs 



(53) 

(54) 
(55) 



L{Q=(t)=0} 



■' ■' 0} 



■E 



ffs 



QI 



This contradicts with the given fact that e^ 



(mox)_|_^s(max) 



<yer + e» = ^'''"""^- 

The second inequality is based on the fact that ]jp™'^^> > 
max{Rf,€s}. 

So far, Zf{t) < zf"'''''\ Vi e [l,F],s e [1,5] for each 
time slot t is proved. 

In conclusion. Lemma [T] is proven. 

Appendix H 
Proof to Theorem[5] 

We prove this theorem by contradiction. 

For each cloud i G [l,F] and each service type s G [1, 5], 
the job requests arrive at time slot t > is rf{t) and the 
earliest time they can depart the queue is t + 1. We show that 
all these jobs depart (by being either scheduled or dropped) 
on or before time t + dg. 

Suppose this is not true, we will come to a contradiction. 
We must have that Qf (t) > for all r G [t + I,. . . ,t + ds] 
(otherwise, all the jobs are scheduled by time t + dg). With 
the queueing law in Eqn. (|4|, we have that 



-. Hence, we have proved that each job of type 
s G [1, S] is either scheduled or dropped with Alg. [T]before its 



maximum response delay ds, if we set e^ — 

Appendix I 
Proof to Theorem|6] 

Since the system status, i.e., the job arrival rf{t) and 
service pricing pf{t) of each type s G [1,5*] of services 
and the operational price f3i{t) at each cloud i G [1,-F], 
changes with ergodic processes, we have that there exists 
a stationary randomized algorithm ||T9l , which dynamically 
decides the VM valuation & pricing (with &™*(i), 7™*(i), 
s™*(t), C*(0)' job scheduling (with ^ii*{t)) & dropping 
(with Df*{t)) and server provisioning (with n™'*{t)) at each 
cloud i, such that the offline optimum of cloud i's individual 
profit ri* can be achieved, together with f| < J2f=i P-iJ+Df* 
and e^ < Xl^i P-tj ~^ -^T ■ Here, a denotes the time averaged 
expectation of variable a{t). 

Based on the derivations of the optimization problem (fTST l 
in and its solution in Sec. |IV| we know that Algorithm [T] 
minimizes the right-hand-side of the drift-plus-penalty (drift- 
minus-utility) inequality in Eqn. ([TtT i at each slot t, with 
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individual profit maximization as the utility, over all possible By summing over the T slots on both sides of the inequality, 

algorithms. Then, we can have that we have that 



T-l 



A(e,(i))-v-[ Yl [sTit)vr{t)-bTmrit)-p.it)nTit)] E{L(e.(r))}-E{L(e.(o))}-v^.^E{[ ^ [srWCW 

™e[l,Afl t=0 me[l,A/] 

''6[i.s] se[i,s] se[i,s] 3e[i,s] 

<B,+ E [Q.^(tKW + ^rw^.]+ E kwp>w] <T-B.-v-j2m E [^rwr/rw-fcrmrw 

se[i,s] sG[i,s] *=o me[i,iv/] 

-\/ E [^r(t)r;r(i)-&r(t)7rw-A(i)nr(i)i -A(t)nr(t)]- E Drmt+ E t^^'Wp^wi]} 

me[i,A/] se[i,S'] se[i,s] 

- E Drit)[QKt) + z!{t)-v-cn 

=e[i.s] +Zr(i)E{[6, -E^■?^W-■°'"W1>^ 

=B, - 1/ • [ E [^r* (*)'?r* (t) - br imr (t) ~ p^ it)nr m 

me[l,A/] 



J = l 



Since E{L(ei(T))} > and £{£(6^(0))} = according 
.5-^ .(-^ to the definition of the Lyapunov function, we have that 

- E Dvmn E ^>(^KW] 

+ E iQKtMit)- E M^;w-A^*w] *=° '"^'^•"i 

se[i,5] je[i,F] - E DUt)£,t+ E KWp'W]]} 

■^ — T — 1 

Taking conditional expectations over queue status Q,{t) on <T ■ Bi - V ■ 2_^E{[ 2^ [sf (t)^'" (i) - C (i)7l" (t) 
both sides the inequality, we have that ^ '"'^ 

E{A(e,(t))ie,(t)}-y-E{[ V [sT(t)fjr(t)-br(mr(t) " p^it)n^{t)] - E Drm,+ E K(*kw]1} 
-A(f)nr(t)]- E Dtmt+ E K(iKW]iie>w} +E E [oKimKw-EMi/w-A-w]} 

se[i,s] se[i,s] t=o sG[i,s] j=i 

<B,-l/-E{[ E [sr(iWr(i)-&r(i)7r(i)-/3.(t)nr(t)] 



F 



me[l,M] 



+ z:(t)E{ie, - E/^-^ w - A^* wi}]- 



- E A"(t)C'+ E ['■-'Wp^WIII^'W} Dividing T • y'on both sides of the above inequality and 

^^'^'^i se[i.s] taking Umitation on T to infinity, we have that 

F F 

+ E [Qt{m{[rt{t)-J2^^tKt)-Dr{t)]\e^m -n, <b,/i/ - n* + E [Qn^,'-E/^-i'--^n 

se[i,s] j=i se[i,s] j=i 



F 



+ znmnes - Em"w - A"(t)]ie,(f)}]. + znes -Y.f^^ ^" 

J=l 3=1 

Next, we take expectations on both sides the inequality, we <R/l/ — O* 

have that ~ 

E{A{e,{t))}~v-E{[ E isT{t)fir{t)-bTmr{t) 

me[l,A/] 



The second inequality comes from the fact that r| — 

EUf^I - ^r < and e, - E^i M?/ - ^f < 0. 
Rearranging the two sides, we have that 



<B,~v-E{[ E [sr(t)c*(i)-c*(t)7r(t)-ft(i)«r(t)] appendix j 

'"^[i.A/] Proof to Theorem!?] 

~ Z^ ^i (i)?i + 2^ Fi (i)Pi(i)IJ} Similar with the proof to the optimality of individual profit 

'"^^'^'^^ sG[i,s] ^jjjj ^jg [Y| ^g jj^yg jjjg following proof to the optimality in 

E\n''f ^^ws\ ^t \ V^ ^*( \ n^*t w\ social welfare for our benchmark algorithm. Since the system 

.s[is] ^ ^^''^ '^^^'^'' '•^•' *^ J°^ ™^^1 ^'(*) ^"'^ ^^1"^^'^^ P"""g ^^'(*) °f 

^ each type s S [1, 5"] of services and the operational price /3i(i) 

+ ^;(i)E{[6, -EMi;(i)- A"(i)]}]- ^^ ^^'^'^ '^1°'^'^ * ^ [1'-^]' changes with ergodic processes, 

j=i we have that there exists a stationary randomized algorithm 
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|fT9l . which dynamically decides the job scheduling (with have that 

fJ.fj{t)) & dropping (with Df*{t)) and server provisioning 

(with nf**(i)) at each cloud i, such that the offline optimum of E{A(e(t))} + V ■ J^ ^i^ J2 Mi)nTit)+ J^ A'WC' 

tiie federation's social welfare JJ* can be achieved, together ig[i p] ^neii m] se[i s] 

with ft* < Y.U f'tl+Df* and ., < EJLi fit;+Dr. Here, _ y ,^(,)p^(i)]} 

a denotes the time averaged expectation of variable a{t). ssiTsi 

Based on the above derivations of the optimization problem <^ i t/ . y^ jgrr y^ B(t)n'^*(t) + Y^ D''*(t)£" 

( |40] l and its solution in Alg. |2] we know that Algorithmic ie[i,F] me[i,M] se[i,s] 

minimizes the right-hand-side of the drift-plus-penalty (drift- y^ «, , s,, ■.-,-, 

minus-welfare) inequality in Eqn. (IJTJ I at each slot t, with y^ 

social welfare maximization as the utility, over all possible ^ 

algorithms. Then, we can have that + ^ ^ [Q^(f)E{K(f) - ^^^;(f) - A"(i)]} 

ie[i,F] se[i,s] j=i 

A(e(t)) + V- J2 [ I] [/3>(f)nr(f)]+ J2 ^''W^' ^ 

ze[i,F] me[i,A/] ss[i,5] + Z^ {t)E{[es ~ }_^ fitj (i) ~ Dr{t)]}]. 

<>eii.s] By summing over the T slots on both sides of the inequality, 

^^+ E [ E [Qnt)rUt) + Z^{t)es]~V J2 rKt)pKt)] we have that 

ie[i,F] se[i,s] 3e[i,s] 



^^.|^F,^£/,'''^^'^"^*^'^^ E{L(e(r))}-E{L(e(o))}-i/.X: E ^il E Mt)nT{t) 



t = ie[l,F] mS[l,M] 



ie[i,F]s=e[i,s]je[i.F] 

^ ^ Drit)lQt{t) + Z^{t)-V-a] 

ie[i,F] se[i,s] 



se[i,s] se[i,s] 

T-l 

<r.B + t/.^ ^ E{[ ^ A(i)nr(t)+ E ^"W^'' 



=B + V^- ^ [ Yl {Mt)nr{t)]+ Yl A'*W?? *=o«e[i.F] ™e[i.A/] «e[i,s] 



iS[l,F] mg[l,Af] 3e[l,S] 



E ^«'(*K(i)] ^^[I'^i 



E ^^^WpK*)]} 



sS[l,S] 



T-l F 



+ E E E [«-wiE{Kw-E^^^w--^«"W]} 
+ E E [q«'wkw-Ea'?w-a^*w] t=o,6[i,F]se[i,s] 

ie[i,F]se[i,s] j=i P 

+ Z^it)E{[e, - E^^i W - A"(i)]}]. 



j=i 



j=i 



Since E{L(e(T))} > and E{L(e(0))} = according to 
Taking conditional expectations over queue status e(<) on the definition of the Lyapunov function, we have that 
both sides the inequality, we have that 



ie[l,F] me[l,M] .^ 

+ ^ A^(t)cr- E rtmmm} ".Jtsi'''^'^''''^'^^^ 



T-l 



se[i,s] se[i,s] 

<B + y. ^ E{[ ^ Mt)nr{t)+ E ^"W^«' <r.B + l/.^ ^ E{[ J2 Ht)nr{t)+ ^ A"(t)Cr 

i:e[l,F] me[l,M] se[l,S] i=0 ie[l,F] me[l,A/] sS[l,S] 

ss[i,s] se[i,s] 

+ E E [Q'^(*wK(i)-EM"(t)-A^*wiie(t)} +E E E [«-w^{[^''w-E^"w--D"W]} 

ie[i,F]ss[i,s] j=i t=o ig[i,F]se[i,s] j=i 

F F 

j=i j=i 

Next, we take expectations on both sides the inequality, we Dividing T • y on both sides and taking limitation on T to 
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infinity, we have that 

-n <B/v -n* 

F 

+ E E [Q^n-E/^'^-Dn 

ie[i,F] se[i,s] j=i 

F 

+ Zi [es " ^ Mij' - Dt*]] 

<B/i/-n*. 

The second inequahty is based on the fact that rf — 
Df* < and e, - Tf_, uf* - L),f* < 0. 






Rearranging the two sides, we have that 

n > n* - Bjv, 

Appendix K 
Proof to Theorem[8] 

We have shown that. Algorithm |2] achieves a social welfare 
with a constant gap to the offline optimum, by minimizing the 
RHS of drift-plus-penalty inequality in Eqn. ( [37| i according to 
the Lyapunov optimization theory fT9l. Hence, if we can prove 
that. Algorithm [T] can also minimize the RHS of Eqn. ( |37] ). 
i.e., maximizing problem dJST l and ( |39] |. we can also prove its 
social welfare optimality. Our intuition of the proof is that, 
when the number of clouds in the federation grows to infinity, 
the gap to the minimum of RHS of Eqn. ( l37b . i.e., the gap 
to the maximum of problem dSST l and ( |39] l, by Algorithm [T] is 
infinitely close to zero. 
- Gap to the minimum of RHS of Eqn. ( 1571 ) with Algorithm\l\ 

As discussed above, problem ( |39] | is only controlled by 
the job drop decisions, i.e., Df{t). Since Algorithm [T] and 
Algorithm |2] have the same decision on job dropping as in 
Eqn. (l30l ) and Eqn. ( l46b . the maximum of problem ( |39] l is also 
achieved by Algorithm [T] Hence, the gap to the minimum of 
RHS of Eqn. ( l37b . by Algorithm [T] only depends on its gap 
to the maximum of problem (|38] |. which is determined by the 
job scheduling and server provisioning decisions. 

We first map the job scheduling and server provisioning 
decisions in Algorithm [2] to an equivalent VM allocation based 
on an idealized double auction scenario. Let each cloud still 
proposes its buy-bid and sell-bid based on the true valuations 
given in Eqn. (EB, (ES, (ESI and ( |26] |. 

With Eqn. (|2TI) . we have that the price of buy-bid for VM 
type m at cloud i is -^ of the maximum weight among all 
jobs at this cloud demanding type-m VMs. With the winner 
determination of our double auction mechanism, 6'™(i) is the 
maximum price of buy-bids from all clouds for type-m VMs. 
Hence, the cloud with buy -bid 0™(^) has the maximum weight 
among all jobs demanding type-m VMs at all clouds. 

According to the definition of < i,„ , Sm > in Eqn. (05]), 
we know that jobs of service type s„i at cloud i,„ has the 
maximum weight for VM type m over all service types at 
each cloud demanding for the same VMs. Hence, cloud im 
proposes the maximum buy -bid price 0"''{t), which is -^ of 
the weight of type-s,„ jobs at cloud im- 

With Eqn. ( |22] |. we have that the price of sell-bid for VM 
type m at cloud i is the larger one between i) y of the 



maximum weight among all jobs at this cloud demanding type- 
m VMs; and ii) the per-VM operational price (3i{t)/C"'- at 
cloud i. With the winner determination of our double auction 
mechanism, ?9"'(t) is the j*^^ lowest price of sell-bids from all 
clouds for type-m VMs. 

If the maximum buy-bid price 6'™(i) is larger than the j*'' 
lowest sell-bid price i?"'(i), we have that i) with an idealized 
double auction, cloud «,„ has a higher buy-bid price than the 
sell-bid such that it can buy all VMs of type m from the 
cloud proposing i?"'(t); ii) with the job scheduling decision in 
Eqn. (|44] |. all VMs of type m are allocated for job scheduling 
at cloud ijn for service type Sm- Hence, the job scheduling 
decision in Algorithm |2] is equivalent to the idealized double 
auction that, the bidder with highest buy-bid price can buy all 
VMs from those sellers with a lower sell-bid price. Fig. |5]gives 
an illustration for the case. In Fig. |5] the buy-bids are sorted 
in descending order while the sell-bids are sorted in ascending 
order Let there are k sell-bids with lower price than that of 
the highest buy-bid 0^{t). Here, k is the maximum number 
of sell-bids, whose prices ??™(i) (j G [l,fc]) are lower than 
that of the highest buy -bid 6l™(i), i.e., 

k = arg max {^"(i) < eT(t)\\/j e [l,i]}. 
ieli.F] 

We know that all the k sellers sell all VMs of type m to the 
cloud with 9™{t), i.e., cloud im- 

However, the idealized cloud cannot give truthfulness guar- 
antee. With our double auction mechanism, in Fig. |5] only 
the {]' — 1) sellers, whose sell-bid prices are no larger than 
the second highest buy-bid price 0^{t), will sell their VMs 
to cloud i,„. Here, j' is the maximum number of sellers with 
sell-bid prices no larger than 9™{t), as defined in Eqn. OTT l. 
Hence, the gap to the maximum of problem d38T l by Algorithm 
[T] is determined by the VMs that are not sold to cloud i„j by 
sellers between j' and k. 

If these VMs of type m are allocated to cloud i^ with 
Algorithm |2] their utility gain in problem (l3Ft is that 
^ Q''-{t) + Zf'-{t) 

J2 [^12 ^2 V/?,/Cf ] ■ Cf iVj" 

= E lOTit)~/3,/Cr]-V-Lfit). 
i=[i'.fc] 
Here, the server provisioning decisions, i.e., n™{t), are re- 
placed by job scheduling decisions, i.e., ufAt), according to 
Eqn. ( |43] ) in Algorithmic 

If these VMs of type m of sellers j G [j' , k] are not traded to 
cloud im based on our double auction mechanism, these VMs 
are either scheduled to serve the jobs with maximum weight 
at its own cloud, if the maximum weight is higher than the 
per-VM operational price, or inactivated otherwise, according 
to the job scheduling decision in Eqn. ( |28] ) of Algorithm [1] 
Hence, the utility gain of these VMs for problem ( [38] l with 
Algorithm [U is that 

J2 WT(i)-P./cT]-v-LT(t). 

So, the gap to the maximum of problem (|38] l by Algorithm 
[T|is the difference between the above utility gains, as follows. 
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V- E [oT{t)^-&Tm-Lr{t), 



(56) 



j^b'M 



which is equivalent to the size of the shadow area in Fig. |5] 
multiplied by V . 

Since i?™(i) > 6'™(i) for each j e [j' + 1, k], we can give 
an upperbound to the gap as follows, 

j = lj',k] 

=v-[[eT{t)^^]i{t)]-L'p{t)+ Y. 

j=[j'+i,fci 



[cw-cwi-^rwi 



<v 

+ 



L7(t) 



Lfit)] 



Y l0T{t)-9T{t)] 

j=[j'+l,ft] 



J = lj',k] 



\t)-eT{t)]-LT{t)] 

(57) 

Since the system is homogenous with the same number of 
servers for each VM type m at each cloud, the value of L™ {t) 
at each cloud j ^ [1, F] is also the same. We use L™ to denote 
that value. Hence, the gap in Eqn. ( fSTJ ) can be rewritten as 

V ■ L'"[[^™+i(i) - ^"pm + [k- j'][eTit) 




\t)] 



Buy-bids: 
descending order 



(58) 



Sell-bids: 
ascending order 



J'-I J' k 

Volume 

Fig. 5. Illustration of the auction. 

We next show that, when the number of clouds scales to 
infinity, the gap in Eqn. ( fSSl l is infinitely close to zero, by 
analyzing the distribution of bidding prices based on the cloud 
number F. 
- Distribution of bidding prices: 

The distribution of the bidding prices should be analyzed in 
order to find an analytical result between the size of the gap 
in Eqn. dSSl) and the number of clouds F. 

Let Qi{t) be the status of cloud i at time slot t. We see 
that the status Qi{t) is a Markov chain with Algorithm [T] 
Define S = {p : Pr(e,(t) = pi 6^(0) = 0) for some t}, then 
Qi{t) is an irreducible Markov chain on state space S with 
&i{0) = 0. This claim is true because i) any state in S is 
reachable from and ii) since Pr(rj (t) = 0) > 0, Vs G [1, 5*], 
the Markov chain can move from <di{t) to in finite time with 
a positive probability. {Qj{t) can be cleared by job scheduling 
or dropping while virtual queue Z^{t) can also be cleared after 
Qf{t) is zero for a constant time, according to the queueing 
laws Eqn. (|2]i and (|4|i) Based on the same reason as above, 
the state is an aperiodic state. With Lemma [T] we know 



that Qf{t) and Zf{t) have finite upperbounds, we can then 
have that the state space S is also finite. In conclusion, the 
Markov chain is irreducible with an aperiodic state and finite 
state space. Hence, the Markov chain is ergodic. 

We know that the prices of buy-bid and sell-bid for each VM 
type m at cloud i are calculated with its current status 6i(t), as 
well as the current operational price (3i{t) (only for sell-bid), 
according Alg. [T] Since the cloud's status follows an ergodic 
process and the operational price is also ergodic according 
to our problem model, we know that the prices of buy-bid 
and sell-bid are also ergodic. Recall that we are proving the 
asymptotic optimality in social welfare by Algorithm [T] under 
homogenous system settings. Hence, we can have that the 
prices of buy-bids and sell-bids at different clouds follow the 
same ergodic process. 

Let the price of buy -bid for VM type m follow a distribution 
£ with continuous density e on the compact interval [0,6]. 

Here, b = 



Qi 



V-gs 



maximum value of — 



'+z. 



where s is the service type with 

-^ among all types in [1, S\ 

with s„j — m. Let the price of sell-bid for VM type m follow 
a distribution H with continuous density h on the compact 
interval [b,6]. Here, b = /3j /C™. Denote the maximum 
and minimum of e and h as follows. 



rmn • / \ ^ n max / \ ^ n 

e = mm e(x) > 0, e — max e(x) > 0, 

-'" " x£[0,b] 



xe[Q,b] 

2 = mm n(x) > 0, h 



ES[b,6] 



max h(x) > 0. 

xe[h,b] 



Then, we can have the following lemma according to 
Lemma 1 in 
Lemma 4: 
1 

e"^°-^(F + 1) 
1 



<E{e,^e,+^}< 

<E{^j+i-^,}< 



^(F+l) 

1 



VjG[l,F-l] 



G[l,i^-ll 



ft™"(F+ 1) 
Here, E{-} denotes the expectation. 
- Asymptotic Optimality in Minimizing the RHS of Eqn 
with Algorithm |7] 

With Lemma ID we can further bound the gap in Eqn 
to the maximum of problem (|38] | by Algorithm [T] as follows, 

V ■ L^Wd^ji^At) ~ ^"(i)l + ffc - A\eT{t) - e^{t)\] 



<V ■L"'[- 



1 



^j'(i)l + ['^-J 
k-f 



+ ■ 



h"^i^(F + l) e™»"(F+l) 

Since each seller between j' and k has a sell-bid price 

between 0™(t) and 9™{t), the sell-bid prices of all these 

sellers resides in an interval which has an expectation no larger 

than „^i„}jj, I -1^ . On the other hand, since the expected interval 

we know 



between each sell-bid is no smaller than 



that the expected interval between the j' seller and the k seller 
should be no smaller than p;;^7|rrjy, which should be still 
no larger than ,„;„ }„,,■, . Hence, we have that 

k-f . 1 



Umax 

^k - f 



< 



rmax 



e™"(_F+ 1) 
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Finally, we can bound the gap to the maximum of problem 
by Algorithm [T] as follows, 

1 vmax / ^min 



'/i'"'"(F+l) e™"(_F'+l)'' 
It is clear that this gap is infinitely close to zero when the 
number of cloud in the federation F grows to infinity. 
- Asymptotic Optimality in Social Welfare of Algorithm\l] 

When the number of clouds in the federation scales to 
infinite large, i.e., F -^ oo, the RHS of Eqn. ( [37] ) is minimized 
in each time slot t with our Algorithm [T] and our double 
auction mechanism. Thus, following the same steps, as above, 
to prove the social welfare optimality of Algorithm |2] which 
also minimizes the RHS of Eqn. ( l37T i in each time slot t, we 
can prove that the time-averaged social welfare achieved with 
our Algorithm [T] is within a constant gap B/V from the offline 
optimum 11*, when F — !• oo, i.e., 

n > n* - B/V, 



